Information Theory

You call CrossEntropyLoss() in every classification model you train. You see "perplexity" in every language model paper. KL divergence shows up in VAEs, distillation, and RLHF, anywhere two probability distributions need to be compared. None of that is a coincidence: all three come from the same handful of ideas.

Claude Shannon wrote those ideas down in 1948 to solve a communication problem: how do you measure the information carried by a signal, and how efficiently can you encode it? Training a neural network turns out to be the same problem in disguise. The model is trying to transmit the correct label through a noisy channel of learned weights, and the loss function is measuring how many bits get lost along the way.

This lesson builds entropy, cross-entropy, and KL divergence from scratch, in that order, so you can see exactly where each one comes from and why it works.

Information Content (Surprise)

The information content of an event x with probability p(x) is I(x) = -log(p(x)): the less likely an event is, the more information you get when it actually happens.

I(x) = -log(p(x))

A coin landing heads carries almost no information. A lottery win carries a lot. Log base 2 gives you bits, natural log gives you nats: same quantity, different units.

Event              Probability    Surprise (bits)
Fair coin heads    0.5            1.0
Rolling a 6        0.167          2.58
1-in-1000 event    0.001          9.97
Certain event      1.0            0.0

A certain event carries zero information, because you already knew it would happen.

Entropy (Average Surprise)

Entropy measures the expected surprise across every outcome of a distribution: weight each event's surprise by how often it happens, then add them up.

H(P) = -sum( p(x) * log(p(x)) )  for all x

A fair coin has the maximum entropy possible for a binary variable, 1 bit. A coin biased 99% toward heads has only 0.08 bits, because you already know what's coming and each flip barely tells you anything new.

Fair coin:    H = -(0.5 * log2(0.5) + 0.5 * log2(0.5)) = 1.0 bit
Biased coin:  H = -(0.99 * log2(0.99) + 0.01 * log2(0.01)) = 0.08 bits

Analogy: think of entropy as how many good guesses a game of twenty questions needs, on average, to land on the answer. A fair coin needs exactly one good guess. A coin that lands heads 99% of the time barely needs a guess at all, you already know what's coming before you ask.

Entropy is the irreducible uncertainty in a distribution. You cannot compress a signal below it.

Cross-Entropy (The Loss Function You Use Every Day)

Cross-entropy measures the average surprise when you use distribution Q to encode events that actually come from distribution P.

H(P, Q) = -sum( p(x) * log(q(x)) )  for all x

P is the true distribution, the labels. Q is your model's predictions. If Q matches P perfectly, cross-entropy equals entropy; any mismatch makes it larger.

Analogy: picture paying a courier by weight, but using someone else's estimate of the package instead of the real scale reading. If their estimate matches the true weight, you pay the fair price, the entropy. If it's off, every shipment costs a little extra, and that extra is exactly what cross-entropy adds on top.

In classification, P is a one-hot vector: the true class has probability 1, everything else 0. That collapses cross-entropy down to:

H(P, Q) = -log(q(true_class))

That's the entire cross-entropy loss formula for classification: push the predicted probability of the correct class as close to 1 as you can.

KL Divergence (Distance Between Distributions)

KL divergence measures how much extra surprise you get from using Q instead of P.

D_KL(P || Q) = sum( p(x) * log(p(x) / q(x)) )  for all x
             = H(P, Q) - H(P)

Cross-entropy is entropy plus KL divergence, and since the entropy of the true distribution stays constant during training, minimizing cross-entropy and minimizing KL divergence are the same move: you're pushing your model's distribution toward the true one.

KL divergence is not symmetric, D_KL(P || Q) != D_KL(Q || P), so it doesn't qualify as a true distance metric.

Analogy: imagine navigating with a map that's slightly wrong. You still get there, but with extra turns and backtracking an accurate map wouldn't need. KL divergence is that overhead, measured in bits: the gap between the shortest path a correct model of the world gives you and the longer one you get by trusting a distribution that's slightly off.

Conditional Entropy

Conditional entropy H(Y|X) measures how much uncertainty about Y remains after you've observed X.

H(Y|X) = H(X,Y) - H(X)

Two extremes:

• If X completely determines Y, H(Y|X) = 0. Knowing X eliminates all uncertainty about Y: temperature in Celsius versus temperature in Fahrenheit is a clean example.
• If X tells you nothing about Y, H(Y|X) = H(Y). Knowing X does not reduce your uncertainty at all: a coin flip tells you nothing about tomorrow's weather.

Conditional entropy is always non-negative and never exceeds H(Y):

0 <= H(Y|X) <= H(Y)

Decision trees lean on exactly this conditional-entropy bound. At each split, the algorithm picks whichever feature X minimizes H(Y|X): the one that removes the most uncertainty about the label Y.

Joint Entropy

Joint entropy H(X,Y) is the entropy of X and Y treated as one combined outcome, over their joint distribution.

H(X,Y) = -sum sum p(x,y) * log(p(x,y))   for all x, y

Key property:

H(X,Y) <= H(X) + H(Y)

Equality holds only when X and Y are independent. Once they share any information, the joint entropy drops below the sum of the individual entropies, and the gap is exactly the mutual information.

The relationships:

• H(X,Y) = H(X) + H(Y|X) = H(Y) + H(X|Y)
• I(X;Y) = H(X) - H(X|Y) = H(Y) - H(Y|X)
• H(X,Y) = H(X) + H(Y) - I(X;Y)

Mutual Information

Mutual information measures how much knowing one variable tells you about another, and unlike KL divergence, it's symmetric: knowing X about Y tells you exactly as much as knowing Y about X.

I(X;Y) = H(X) - H(X|Y)
       = H(Y) - H(Y|X)
       = H(X) + H(Y) - H(X,Y)
       = sum sum p(x,y) * log(p(x,y) / (p(x) * p(y)))

If X and Y are independent, mutual information is zero: knowing one tells you nothing about the other. If they're perfectly correlated, it equals the entropy of either variable.

Properties:

• I(X;Y) >= 0 always, since you never lose information by observing something.
• I(X;Y) = 0 if and only if X and Y are independent.
• I(X;Y) = I(Y;X): it's symmetric, unlike KL divergence.
• I(X;X) = H(X): a variable shares all its information with itself.

Analogy: think of mutual information as how much a friend's restaurant opinion actually tells you. If their taste has nothing to do with yours, hearing it adds nothing, zero mutual information. If their taste always tracks yours exactly, their opinion tells you everything you'd have learned by trying the place yourself.

Mutual information for feature selection. High mutual information between a feature and the target means the feature is useful; low mutual information means it's noise. Mutual information gives you a principled way to rank features:

1. For each feature X_i, compute I(X_i; Y) against the target Y.
2. Rank features by MI score.
3. Keep the top k.

This catches any relationship between a feature and the target: linear, nonlinear, monotonic, or not. Correlation only catches the linear ones; MI catches all of them.

Label Smoothing and Cross-Entropy

Standard classification uses hard targets, like [0, 0, 1, 0]: the true class gets probability 1, everything else gets 0. Label smoothing replaces that with a soft target:

soft_target = (1 - epsilon) * hard_target + epsilon / num_classes

With epsilon = 0.1 and 4 classes:

• Hard target: [0, 0, 1, 0]
• Soft target: [0.025, 0.025, 0.925, 0.025]

From an information theory angle, label smoothing raises the entropy of the target distribution. Hard one-hot targets have zero entropy: no uncertainty at all. Soft targets carry positive entropy instead.

Why this helps:

• It keeps the model from driving logits to extreme values, since matching a one-hot target exactly under cross-entropy would need infinite logits.
• It acts as regularization: the model can never claim 100% confidence.
• It improves calibration, so predicted probabilities track true uncertainty more closely.
• It shrinks the gap between training and inference behavior.

The cross-entropy loss with label smoothing becomes:

L = (1 - epsilon) * CE(hard_target, prediction) + epsilon * H_uniform(prediction)

The second term penalizes predictions that stray far from uniform, which works as direct regularization on confidence.

Why Cross-Entropy Is THE Classification Loss

Three perspectives, same conclusion.

Information theory view. Cross-entropy measures how many bits you waste by using your model's distribution instead of the true one. Minimize it and your model becomes the most efficient encoder of reality it can be.

Maximum likelihood view. For N training samples with true classes y_i:

Likelihood     = product( q(y_i) )
Log-likelihood = sum( log(q(y_i)) )
Negative log-likelihood = -sum( log(q(y_i)) )

That last line is cross-entropy loss. Minimizing cross-entropy is the same as maximizing the likelihood of the training data under your model.

Gradient view. The gradient of cross-entropy with respect to the logits is just (predicted - true) — clean, stable, and fast to compute, which is exactly why it pairs so well with softmax.

Bits vs Nats

The only difference is the log base.

log base 2   -> bits      (information theory tradition)
log base e   -> nats      (machine learning convention)
log base 10  -> hartleys  (rarely used)

1 nat = 1/ln(2) bits = 1.4427 bits. PyTorch and TensorFlow use natural log (nats) by default.

Perplexity

Perplexity is the exponential of cross-entropy: the effective number of equally likely choices the model is uncertain between at each step.

Perplexity = 2^H(P,Q)   (if using bits)
Perplexity = e^H(P,Q)   (if using nats)

A language model with perplexity 50 is, on average, as confused as if it had to pick uniformly from 50 possible next tokens. Lower is better.

Analogy: think of perplexity as a multiple-choice question with a variable number of options. A model with perplexity 4 is choosing among roughly 4 plausible next words, about as hard as a well-written quiz question. A model with perplexity 4000 might as well be guessing blind from the whole dictionary.

GPT-2 landed around perplexity 30 on common benchmarks. Modern models sit in the single digits for well-represented domains.

Step 1: Information content and entropy

import math
 
def information_content(p, base=2):
    if p <= 0 or p > 1:
        return float('inf') if p <= 0 else 0.0
    return -math.log(p) / math.log(base)
 
def entropy(probs, base=2):
    return sum(
        p * information_content(p, base)
        for p in probs if p > 0
    )
 
fair_coin = [0.5, 0.5]
biased_coin = [0.99, 0.01]
fair_die = [1/6] * 6
 
print(f"Fair coin entropy:   {entropy(fair_coin):.4f} bits")
print(f"Biased coin entropy: {entropy(biased_coin):.4f} bits")
print(f"Fair die entropy:    {entropy(fair_die):.4f} bits")

Step 2: Cross-entropy and KL divergence

def cross_entropy(p, q, base=2):
    total = 0.0
    for pi, qi in zip(p, q):
        if pi > 0:
            if qi <= 0:
                return float('inf')
            total += pi * (-math.log(qi) / math.log(base))
    return total
 
def kl_divergence(p, q, base=2):
    return cross_entropy(p, q, base) - entropy(p, base)
 
true_dist = [0.7, 0.2, 0.1]
good_model = [0.6, 0.25, 0.15]
bad_model = [0.1, 0.1, 0.8]
 
print(f"Entropy of true dist:     {entropy(true_dist):.4f} bits")
print(f"CE (good model):          {cross_entropy(true_dist, good_model):.4f} bits")
print(f"CE (bad model):           {cross_entropy(true_dist, bad_model):.4f} bits")
print(f"KL divergence (good):     {kl_divergence(true_dist, good_model):.4f} bits")
print(f"KL divergence (bad):      {kl_divergence(true_dist, bad_model):.4f} bits")

Step 3: Cross-entropy as classification loss

def softmax(logits):
    max_logit = max(logits)
    exps = [math.exp(z - max_logit) for z in logits]
    total = sum(exps)
    return [e / total for e in exps]
 
def cross_entropy_loss(true_class, logits):
    probs = softmax(logits)
    return -math.log(probs[true_class])
 
logits = [2.0, 1.0, 0.1]
true_class = 0
 
probs = softmax(logits)
loss = cross_entropy_loss(true_class, logits)
 
print(f"Logits:      {logits}")
print(f"Softmax:     {[f'{p:.4f}' for p in probs]}")
print(f"True class:  {true_class}")
print(f"Loss:        {loss:.4f} nats")
print(f"Perplexity:  {math.exp(loss):.2f}")

Step 4: Cross-entropy equals negative log-likelihood

import random
 
random.seed(42)
 
n_samples = 1000
n_classes = 3
true_labels = [random.randint(0, n_classes - 1) for _ in range(n_samples)]
model_logits = [[random.gauss(0, 1) for _ in range(n_classes)] for _ in range(n_samples)]
 
ce_loss = sum(
    cross_entropy_loss(label, logits)
    for label, logits in zip(true_labels, model_logits)
) / n_samples
 
nll = -sum(
    math.log(softmax(logits)[label])
    for label, logits in zip(true_labels, model_logits)
) / n_samples
 
print(f"Cross-entropy loss:      {ce_loss:.6f}")
print(f"Negative log-likelihood: {nll:.6f}")
print(f"Difference:              {abs(ce_loss - nll):.2e}")

Step 5: Mutual information

def mutual_information(joint_probs, base=2):
    rows = len(joint_probs)
    cols = len(joint_probs[0])
 
    margin_x = [sum(joint_probs[i][j] for j in range(cols)) for i in range(rows)]
    margin_y = [sum(joint_probs[i][j] for i in range(rows)) for j in range(cols)]
 
    mi = 0.0
    for i in range(rows):
        for j in range(cols):
            pxy = joint_probs[i][j]
            if pxy > 0:
                mi += pxy * math.log(pxy / (margin_x[i] * margin_y[j])) / math.log(base)
    return mi
 
independent = [[0.25, 0.25], [0.25, 0.25]]
dependent = [[0.45, 0.05], [0.05, 0.45]]
 
print(f"MI (independent): {mutual_information(independent):.4f} bits")
print(f"MI (dependent):   {mutual_information(dependent):.4f} bits")

Here's the same math with NumPy, the way you'll actually use it in practice:

import numpy as np
 
def np_entropy(p):
    p = np.asarray(p, dtype=float)
    mask = p > 0
    result = np.zeros_like(p)
    result[mask] = p[mask] * np.log(p[mask])
    return -result.sum()
 
def np_cross_entropy(p, q):
    p, q = np.asarray(p, dtype=float), np.asarray(q, dtype=float)
    mask = p > 0
    return -(p[mask] * np.log(q[mask])).sum()
 
def np_kl_divergence(p, q):
    return np_cross_entropy(p, q) - np_entropy(p)
 
true = np.array([0.7, 0.2, 0.1])
pred = np.array([0.6, 0.25, 0.15])
print(f"Entropy:    {np_entropy(true):.4f} nats")
print(f"Cross-ent:  {np_cross_entropy(true, pred):.4f} nats")
print(f"KL div:     {np_kl_divergence(true, pred):.4f} nats")

You just built, by hand, what torch.nn.CrossEntropyLoss() does internally. That's why the loss curve drops during training: your model's predicted distribution is closing the gap on the true distribution, measured in nats of wasted information.

1. Compute the entropy of the English alphabet assuming uniform distribution (26 letters). Then estimate it using actual letter frequencies. Which is higher and why?

2. A model outputs logits [5.0, 2.0, 0.5] for a sample with true class 1. Compute the cross-entropy loss by hand, then verify with your cross_entropy_loss function. What logits would give zero loss?

3. Show that KL divergence is not symmetric. Pick two distributions P and Q and compute D_KL(P || Q) and D_KL(Q || P). Explain why they differ.

4. Build a function that computes perplexity for a sequence of token predictions. Given a list of (true_token_index, predicted_logits) pairs, return the perplexity of the sequence.

• Shannon 1948: A Mathematical Theory of Communication - the original paper, still readable
• Visual Information Theory (Chris Olah) - best visual explanation of entropy and KL divergence
• PyTorch CrossEntropyLoss docs - how the framework implements what you just built