Information theory fundamentals

This page is the “theory” version of the topic; for applied RL framing, see the Hugging Face Deep RL notes. This revision adds Python, tables, images, and footnotes so you can see how math-heavy posts look when they’re not pretending to be short.¹

Shannon entropy

Shannon’s central question (1948): how do you quantify information? His answer: information is the reduction of uncertainty. If I tell you something you already knew, I’ve given you zero information. If I tell you something surprising, I’ve given you a lot.

Formally, the entropy of a random variable \(X\) with possible outcomes \(x_1, \dots, x_n\):

\[H(X) = -\sum_{i} p(x_i) \log_2 p(x_i)\]

Entropy is maximized when all outcomes are equally likely (maximum uncertainty) and minimized (zero) when the outcome is deterministic. For a fair coin: \(H = -2 \times 0.5 \log_2(0.5) = 1\) bit. For a biased coin (99% heads): \(H \approx 0.08\) bits. The biased coin carries less information per flip because you already know what’s probably going to happen.

Quick reference — Bernoulli entropy

Takeaway: “surprise” and “information” are the same currency — measured in bits when you use \(\log_2\).²

Tiny Python sanity check

import math
from typing import Iterable

def entropy_bits(p: Iterable[float]) -> float:
    """Shannon entropy in bits for a discrete distribution."""
    return -sum(
        pi * math.log2(pi)
        for pi in p
        if pi > 0.0
    )

print(entropy_bits([0.5, 0.5]))          # 1.0
print(entropy_bits([0.99, 0.01]))        # ~0.0808

Cross-entropy

The cross-entropy between a true distribution \(p\) and a model distribution \(q\):

\[H(p, q) = -\sum_{i} p(x_i) \log q(x_i)\]

This is the loss function used in virtually all classification tasks in ML. When \(q = p\), cross-entropy equals entropy (the minimum). When \(q\) diverges from \(p\), cross-entropy increases. Training minimizes cross-entropy, which pushes the model’s predicted distribution toward the true distribution.

Numerical hygiene (footnote-sized lecture)

In code, you rarely compute log(0) directly — you clamp, or you use the built-in stabilized loss.³

import torch
import torch.nn.functional as F

# logits: raw scores from the last linear layer; target: class indices
def ce_loss(logits: torch.Tensor, target: torch.Tensor) -> torch.Tensor:
    return F.cross_entropy(logits, target)  # log-softmax + NLL, stabilized internally

KL divergence

The Kullback-Leibler divergence measures how different \(q\) is from \(p\):

\[D_{KL}(p \| q) = \sum_{i} p(x_i) \log \frac{p(x_i)}{q(x_i)} = H(p, q) - H(p)\]

Since \(H(p)\) is constant with respect to \(q\), minimizing cross-entropy is equivalent to minimizing KL divergence. This is why cross-entropy is used as a loss: it’s a proxy for “how wrong is the model.”

Important: KL divergence is NOT symmetric. \(D_{KL}(p \| q) \neq D_{KL}(q \| p)\). This asymmetry has practical consequences:

• Mode-covering (\(D_{KL}(p \| q)\)): the model tries to cover all modes of \(p\), even at the cost of spreading probability mass to unlikely regions. This is what variational inference typically minimizes.
• Mode-seeking (\(D_{KL}(q \| p)\)): the model collapses to a single mode of \(p\), ignoring others. This is what GANs tend to do.

Nested blockquote, because this point confuses everyone at least once:

Remember: KL is an expectation under \(p\) of log-ratios.

If you swap which distribution sits “under the expectation,” you get a different objective — not “the same thing but tweaked.”

Mutual information

The mutual information between two variables \(X\) and \(Y\):

\[I(X; Y) = H(X) - H(X|Y) = H(Y) - H(Y|X)\]

It measures how much knowing one variable reduces uncertainty about the other. If \(X\) and \(Y\) are independent, \(I(X;Y) = 0\). If knowing \(X\) completely determines \(Y\), then \(I(X;Y) = H(Y)\).

This shows up in feature selection (which features carry the most information about the target), representation learning (InfoNCE loss), and decision trees (information gain is mutual information between a feature and the class label).

Figure: classic RL diagram (caption pattern)

Cross-linking visuals helps when you’re bouncing between “information theory” and “agents in environments”:

The RL loop — states, actions, rewards — same picture as in the HF RL notes, different surrounding prose.

Legacy screenshot kept around to test PNG + caption styling on long pages.

Why this matters

Shannon’s framework provides a rigorous language for talking about learning. Training a model is reducing entropy. Overfitting is memorizing noise instead of structure. Compression and prediction are the same problem viewed from different angles (a good predictor is a good compressor, and vice versa). Information theory doesn’t tell you how to learn, but it tells you what learning means.

Checklist: translating theory to debugging questions

1. Calibration: Are my predicted probabilities meaningful — or just ranking scores?⁴
2. Bottlenecks: Is mutual information between representation and label actually non-trivial?
3. Regularization: Am I penalizing capacity or penalizing noise fitting — different fixes.

# Not information theory — just the vibe of "measure everything"
python - <<'PY'
import math
for p in [0.5, 0.9, 0.99]:
    print(p, -p*math.log2(p) - (1-p)*math.log2(1-p))
PY

Table stress (entropy, code, dollars)

Footer: all three columns mix TeX and backticks.

Footer: literal \$ for fake prices + long third column.

Further reading (external): MacKay’s Information Theory, Inference, and Learning Algorithms (free PDF) — famously readable. For ML-heavy treatment: Bishop’s Pattern Recognition and ML (chapters on entropy/KL in discrete settings).