Module 4 · suggested week 4

Probability, Entropy, and Information

How science measures uncertainty.

~3 focused hoursBuilds on module 1 (advisory)

0 of 7 sections read

What you will be able to do

Reason with conditional probability and Bayes, read −Σ p log p in its physics, biology and ML forms, and say precisely what each version measures and assumes.

\sum

Summation (capital sigma)

\log

Logarithm

\mathbb{E}

Expectation (expected value)

P(A \mid B)

Conditional probability

\mu

\sigma

Sigma (lowercase)

\propto

Proportional to

e^{x}

Exponential

\mathcal{L}

Loss function (calligraphic L)

\in

Element of (set membership)

The breakthrough: entropy is a weighted loop over surprise

Recognition

Surprise, averaged

Define the surprise of an outcome with probability $p$ as $-\log_2 p$ bits. A certain event ( $p = 1$ ) carries 0 bits of surprise. A fair coin flip carries 1 bit. A 1-in-1024 event carries 10 bits. Entropy is the expected surprise: loop over every outcome, multiply its surprise by its probability, and add up.

H(p) = \sum_i p_i \cdot \big(-\log_2 p_i\big)

function entropy(probs) {
  let H = 0;
  for (const p of probs) {
    if (p > 0) H += p * -Math.log2(p);  // probability × surprise
  }
  return H; // in bits
}
entropy([0.5, 0.5]);          // 1 bit (fair coin)
entropy([0.25, 0.25, 0.25, 0.25]); // 2 bits
entropy([0.99, 0.01]);        // ≈ 0.08 bits (nearly certain)

Entropy & cross-entropy

p weight · A0.60p weight · C0.20p weight · G0.10p weight · T0.10q weight · A0.25q weight · C0.25q weight · G0.25q weight · T0.25

Entropy H(p)1.571 bits

Maximum possible (uniform)2.000 bits

Cross-entropy H(p,q)2.000 bits

KL(p‖q) = gap0.429 bits

$H(p)$ is the average surprise, $-\log p_i$ , when outcomes really follow $p$ . Cross-entropy $H(p,q)$ is the average surprise when reality is $p$ but you predict with $q$ . It is never below $H(p)$ , and the gap is the KL divergence. Language-model training minimises cross-entropy between real next tokens and the model’s predictions. The labels A, C, G, T are DNA bases, as in a sequence logo, but the arithmetic does not care what the outcomes are.

Loading interactive…

Drag the bars of distribution p. Entropy peaks when outcomes are equally likely and hits zero when one is certain. Then set a model distribution q and watch cross-entropy H(p, q) rise above H(p) by exactly the KL divergence.

Once you see entropy as Σ-of-probability-times-surprise, a large share of the equations in ML papers become readable. The expression $\sum_i p_i \log p_i$ (or its continuous cousin $\int p(x) \log p(x)\,dx$ ) underlies entropy, cross-entropy, KL divergence, mutual information, log-likelihood and the Gibbs entropy of physics. The difference between them is which distributions go in the $p$ and the $\log$ , and what those distributions describe.

Probability, distributions and expected value

A probability distribution assigns a number between 0 and 1 to each possible outcome, with the numbers summing to 1. For a language model's next token, the outcomes are the vocabulary (often 50,000–200,000 tokens) and the distribution is the model's output.

Probabilities can be interpreted in two broad ways. Long-run frequencies ("this coin lands heads 50% of the time") or degrees of belief ("I'm 70% confident this paper replicates"). The mathematics is the same; the interpretation matters for what claims you can make. Many debates about statistics are really debates about which interpretation is in play.

Expected valueEstablished

\mathbb{E}[X] = \sum_x x\,P(X = x)

The probability-weighted average of the outcomes. Loop over every possible value, multiply it by its probability, add.

// Expected value of a fair die
const outcomes = [1, 2, 3, 4, 5, 6];
const E = outcomes.reduce((s, x) => s + x * (1 / 6), 0); // 3.5

Does not say: An expected value need not be a possible outcome (no die shows 3.5), and it can hide enormous variance. Heavy-tailed distributions can have expectations dominated by rare events, or no finite expectation at all.

$\mathbb{E}$ (blackboard-bold E) appears constantly in ML: $\mathbb{E}_{x \sim D}[\,\ell(x)\,]$ reads "the average loss, with $x$ drawn from the data distribution $D$ ". The $\sim$ means "sampled from". In practice it is estimated by averaging over a finite sample: the training set or a batch. The expectation is the ideal; the sample average is what gets computed. The gap between them is where overfitting lives.

Two numbers summarise many distributions: the mean $\mu$ (centre) and the standard deviation $\sigma$ (spread). The normal (Gaussian) distribution is fully described by those two. It appears so often because sums of many small independent effects tend toward it (the central limit theorem). Many real-world quantities are not normal, though: word frequencies, city sizes and wealth are heavy-tailed, and treating them as Gaussian badly underestimates extremes.

Go deeper: Deeper: sampling

Every empirical claim comes from a sample. Estimates from samples have uncertainty that shrinks roughly as $1/\sqrt{n}$ , so to halve the error you need four times the data. A benchmark with 200 questions can't distinguish 71% from 73% accuracy with much confidence. When a model "beats" another by one point on a small benchmark, ask for the sample size.

Conditional probability and Bayesian updating

$P(A \mid B)$ reads "the probability of $A$ given $B$ ": restrict attention to the cases where $B$ happened, then ask how often $A$ also happened. The vertical bar is a filter.

The single most common reasoning error in science communication is confusing $P(A \mid B)$ with $P(B \mid A)$ . The probability that a person who tests positive is sick is not the probability that a sick person tests positive. Bayes' rule connects them:

Bayes' ruleEstablished

P(H \mid E) = \frac{P(E \mid H)\,P(H)}{P(E)}

Belief in hypothesis H after seeing evidence E = how well H predicted E, times your prior belief in H, divided by how expected E was overall.

// Rare condition (1%), test 90% sensitive, 9% false positives
const prior = 0.01, sens = 0.90, fpr = 0.09;
const pE = sens * prior + fpr * (1 - prior);
const posterior = (sens * prior) / pE; // ≈ 0.092, about 9%

Does not say: The rule is a theorem. The inputs (especially the prior P(H)) are judgments, and garbage priors give garbage posteriors with perfect arithmetic.

Inspect symbol by symbol →

Bayesian updating

■ H true, positive (9)■ H false, positive (89)■ H true, missed (1)■ H false, negative (901)

Prior

P(H)

1.0%Sensitivity

P(E\mid H)

90%False-positive rate

P(E\mid \neg H)

9.0%

Posterior

P(H\mid E)

9.2%

Bayes factor (evidence strength)×10.0

Prior → posterior1.0% → 9.2%

Bayes’ rule, $P(H\mid E) = \dfrac{P(E\mid H)\,P(H)}{P(E)}$ , counted out. Among everyone who tests positive (filled squares), what fraction actually has $H$ ? When the prior is small, false positives from the large “not-H” group can swamp the true ones. Read “H” as “this research hypothesis is true” and the same arithmetic explains why surprising single studies often fail to replicate.

Loading interactive…

Natural frequencies make Bayes visible: imagine 1,000 people, colour who has the condition and who tests positive. Lower the base rate and watch a "90% accurate" test produce mostly false alarms.

Shannon entropy: information as reduced uncertainty

Shannon entropyEstablished

H(X) = -\sum_i p_i \log p_i

The average number of bits (with log base 2) needed to communicate an outcome drawn from distribution p, using the best possible code.

Does not say: Defined for a probability distribution, not for a "message" or a "system" in itself. Entropy of English depends on what model of English supplies the p. The minus sign makes it positive, because logs of probabilities are ≤ 0.

Inspect symbol by symbol →

Established

Shannon's source coding theorem (1948): no lossless code can use fewer than $H$ bits per symbol on average, and codes can get arbitrarily close to $H$ . Entropy is a hard limit on compression. This is why "prediction is compression": a better predictive model of text assigns higher probability to what actually comes next, which means fewer bits to encode it.

Source: Shannon 1948, A Mathematical Theory of Communication

Information, in Shannon's sense, is about uncertainty, not meaning. A random string of letters has maximal entropy and no meaning; a profound sentence can have low entropy under a good model of language. Shannon said explicitly that the semantic aspects of communication are irrelevant to the engineering problem. When essays slide from "information-theoretic" to "meaningful", they have changed subjects.

Cross-entropy and likelihood: the loss inside every LLM

Cross-entropyEstablished

H(p, q) = -\sum_i p_i \log q_i

The average surprise you experience if the world follows p but you predict with q. It is always at least H(p); the excess is the cost of a wrong model.

Inspect symbol by symbol →

Cross-entropy decomposesEstablished

H(p, q) = H(p) + D_{\mathrm{KL}}(p \,\|\, q)

Cross-entropy = the irreducible uncertainty of the data + the KL divergence, which measures how far your model is from the truth. Training can only shrink the second term.

Does not say: This decomposition is why a scaling-law asymptote L∞ is often read as the "entropy of the data". That reading is plausible but an interpretation: the true p of natural language is unknown.

For a language model, at each position the "true" distribution $p$ puts all its probability on the token that actually came next. The sum then collapses to a single term: the loss is just minus the log of the probability the model gave the correct token.

Per-token language-model lossEstablished

\mathcal{L} = -\frac{1}{T}\sum_{t=1}^{T} \log q_\theta(x_t \mid x_{<t})

Average, over every position t in the text, of minus the log-probability the model (with parameters θ) assigned to the actual next token given everything before it.

import math
def lm_loss(probs_of_correct_token):
    # one probability per position: q(x_t | x_<t)
    return -sum(math.log(p) for p in probs_of_correct_token) / len(probs_of_correct_token)

lm_loss([0.5, 0.9, 0.05])  # ≈ 1.27 nats

Does not say: This is the generic objective of pretraining. Production training adds many details (data mixtures, regularisation, later fine-tuning objectives) that this line does not describe.

Token probabilities

The transformer was introduced in a paper titled “Attention Is All You▁

Need

95.9%z=9.1

Want

1.9%z=5.2

Get

0.9%z=4.4

Have

0.6%z=4.1

Know

0.3%z=3.2

Require

0.2%z=2.9

need

0.1%z=2.4

Deserve

0.1%z=1.6

Temperature

T

1.00

Entropy of next-token distribution0.33 bits

Loss if true token is “Need”0.042 nats

T → 0 behaves likeargmax (greedy)

A model outputs one score (logit) $z_i$ per vocabulary token. Softmax, $p_i = e^{z_i/T} / \sum_j e^{z_j/T}$ , turns the scores into probabilities that sum to 1. Temperature $T$ rescales the scores before the exponent: low T sharpens the distribution, high T flattens it. These eight logits were written by hand for teaching, and a real vocabulary has tens of thousands of tokens. The training loss on this step would be $-\ln p(\text{“ Need”})$ .

Loading interactive…

A toy next-token distribution. Adjust temperature: low temperature sharpens toward the top token (lower entropy), high temperature flattens. Sample a few tokens and see the per-token loss −log q of each one.

Minimising this loss is the same as maximising likelihood: making the observed training text as probable as possible under the model. Multiplying probabilities over a long text gives vanishingly small numbers (a Π loop), so everyone takes the log, which turns the Π into a Σ. That is the Module 1 log trick in production.

Perplexity, often reported instead, is $e^{\mathcal{L}}$ (or $2^{\mathcal{L}}$ in bits). Read it as the effective number of equally likely tokens the model is choosing between. Perplexity 10 means "as uncertain as a fair 10-sided die" on average. Compare only across identical tokenizers: a different tokenizer changes what counts as one token.

One formula, four sciences

Here is the payoff, and the trap. The expression $-\sum p \log p$ appears in at least four places you will meet in papers:

−Σ p log p across fields: same form, different claims

		Definition	What p describes	Key assumptions
Gibbs entropy (statistical physics)	$S = -k_B \sum_i p_i \ln p_i$	Probability of each microstate of a physical system	An ensemble of microstates; equilibrium for the standard results; $k_B$ gives thermodynamic units (J/K)	Thermodynamic: tied to heat, temperature, the second law
Shannon entropy (communication)	$H = -\sum_i p_i \log_2 p_i$	Probability of each symbol from a source	A known source distribution (often assumed stationary)	Bits needed to encode outcomes; a limit on compression
Sequence-logo entropy (genomics)	$H_j = -\sum_{b} f_{b,j} \log_2 f_{b,j}$ per position $j$	Observed frequency of each base or amino acid at one position across aligned sequences	A good alignment; enough sequences; positions treated independently	Low entropy = conserved position, often functionally important. Logos plot $\log_2 4 - H_j$ as letter height
Cross-entropy loss (ML)	$-\sum_i p_i \log q_i$	p = target labels; q = model prediction	Training data stands in for the true distribution	An objective to minimise; equals log-likelihood up to sign
Per-token LM loss	$-\frac{1}{T}\sum_t \log q_\theta(x_t \mid x_{<t})$	Model probability of each actual next token	Fixed tokenizer; a dataset assumed to be representative	Average surprise per token; estimates cross-entropy to the text distribution

Established

Shannon remarked in 1948 that his $H$ has the same form as entropy in statistical mechanics. E. T. Jaynes (1957) went further, arguing that statistical mechanics can be derived as inference: maximising Shannon entropy subject to known constraints yields the Boltzmann distribution. This is a substantive mathematical unification of the methods.

Analogy

"An LLM's loss is literally thermodynamic entropy" or "training a model reduces the entropy of the universe" conflate the forms. Cross-entropy loss has no units of J/K, involves no heat bath and obeys no second law. (The physical computer running training does dissipate heat, but that is a separate, physical fact.) Same equation, different referent.

Correlation, causation and likelihood in practice

Correlation measures how two variables move together, from −1 to +1. It says nothing about why. Three classic alternatives to "A causes B": B causes A (reverse causation), C causes both (confounding), or coincidence plus selection (you only looked at cases where both appeared).

The gold standard for causation is intervention: change A while holding everything else fixed (randomisation does this statistically) and see if B changes. In ML research, ablations play this role: remove a component and measure the effect. In interpretability, causal interventions on activations (patching, steering) are how researchers move from "this feature correlates with that behaviour" to "this feature drives it".

Established

A correlation between model scale and benchmark score, measured across models from different labs, is confounded by everything else that differs: data, training recipe, tokenizer, evaluation harness. Controlled scaling studies vary one factor within a single family for exactly this reason.

Connections, with their levels

PhysicsLevel 2 · Shared math

Gibbs entropy and Shannon entropy

Identical mathematical form; Jaynes showed statistical mechanics can be derived by maximising Shannon entropy under constraints.

Why this level: A rigorous shared formalism and a productive methodological link. Thermodynamic and informational entropy still describe different things and carry different units.

PhysicsLevel 2 · Shared math

Softmax and the Boltzmann distribution

Softmax with temperature, exp(z/T)/Σexp(z/T), has the same form as the Boltzmann distribution exp(−E/kT)/Z, with logits playing the role of negative energies.

Why this level: Exact mathematical correspondence, which is why ML borrowed the word "temperature". There is no thermal bath in a language model.

BiologyLevel 3 · Tested prediction

Conservation in sequence alignments

Per-position entropy across aligned DNA or protein sequences flags conserved, often functionally constrained, positions.

Why this level: Low-entropy positions are routinely used to predict functional importance, and those predictions are tested experimentally (mutagenesis). The prediction comes from evolutionary constraint, not from the formula itself.

AI / MLLevel 3 · Shared mechanism

Prediction as compression

Minimising per-token cross-entropy is equivalent to building a better compressor of the training text (via arithmetic coding).

Why this level: A theorem-backed equivalence, not an analogy. What it implies about "understanding" is a separate, open question.

Skeptic’s corner: Probability and information claims

Uncertainty language is where precise science most often turns into confident nonsense. Most errors are one of a handful of swaps.

Has P(A|B) been swapped for P(B|A)? What is the base rate?
Entropy of which distribution, estimated from what sample, in which units?
Is information-theoretic "information" being treated as meaning?
Is a correlation across uncontrolled comparisons being read as cause?
Are loss or perplexity numbers compared across different tokenizers or datasets?

Watch & explore

Video31 min3Blue1Brown

Solving Wordle using information theory

Shannon entropy built from scratch on a game you already understand: bits as halvings of the possibility space.

Watch for: The formula −Σ p log p appearing as "expected information". It is the same expression as in the Entropy interactive.

Verified: YouTube oEmbed title+channel check, 2026-10-30

Video15 min3Blue1Brown

Bayes theorem, the geometry of changing beliefs

Bayes as restricting the space of possibilities, drawn as areas.

Watch for: The "Steve the librarian" example. It is a base-rate lesson in disguise.

Verified: YouTube oEmbed title+channel check, 2026-10-30

Video21 min3Blue1Brown

The medical test paradox, and redesigning Bayes' rule

Bayes factors as "how much should this evidence move me". That is the most useful form for reading claims.

Watch for: Odds × Bayes factor. Use it on any "new study shows" headline.

Verified: YouTube oEmbed title+channel check, 2026-10-30

CourseSanta Fe Institute Complexity Explorer

Introduction to Information Theory (tutorial)

A short, gentle tutorial from bits to entropy, with complexity-science applications.

Watch for: Entropy defined over a probability distribution. Always ask "entropy of which distribution?"

Verified: HTTP 200 + page content checked, 2026-10-30

InteractiveBrown University

Seeing Theory

Visual, interactive probability and statistics, from chance events to Bayesian inference.

Watch for: The chapter on Bayesian inference, as a second view on the Bayes interactive.

Verified: HTTP 200 + page content checked, 2026-10-30

PaperClaude E. Shannon

A Mathematical Theory of Communication (1948)

The origin of H = −Σ p log p. Sections 1 to 6 are readable with Module 4 in hand.

Watch for: Shannon’s own remark that his H has the same form as entropy in statistical mechanics. He notes the form, not a shared mechanism.

Verified: HTTP 200 + page content checked, 2026-10-30

Video27 minVeritasium

The Most Misunderstood Concept in Physics

Thermodynamic entropy as a count of arrangements, not "disorder". The right setup for comparing it with Shannon entropy.

Watch for: Where energy spreading becomes probability. Hold that in mind when Module 4 puts Boltzmann and Shannon side by side.

Verified: YouTube oEmbed title+channel check, 2026-10-30

CourseMIT OpenCourseWare

6.041SC Probabilistic Systems Analysis

Short recitation videos on conditional probability, Bayes and expectation. Use the ones that match a gap.

Watch for: Conditioning as shrinking the sample space.

Verified: HTTP 200 + page content checked, 2026-10-30

CourseKhan Academy

Statistics and probability

Gap repair for distributions, sampling, significance and study design.

Watch for: The study-design unit. Observational studies vs experiments is the correlation/causation line.

Verified: HTTP 200 + page content checked, 2026-10-30

Video24 min3Blue1Brown

Why π is in the normal distribution (beyond integral tricks)

Optional depth on the bell curve, μ and σ, and what the normalising constant is for.

Watch for: e^{−x²}: the shape is set by the exponent, and everything else is bookkeeping so the total probability is 1.

Verified: YouTube oEmbed title+channel check, 2026-10-30

Research & references

Peer-reviewedC. E. Shannon · 1948 · Bell System Technical Journal
A Mathematical Theory of Communication
Read sections 1–6. Limitation: defines information without meaning, by design. Don’t cite it for claims about semantics.
PreprintKaplan et al. · 2020 · arXiv:2001.08361
Scaling Laws for Neural Language Models
The loss on its y-axes is exactly the per-token cross-entropy above. Limitation: loss in nats on their data and tokenizer, not comparable across setups.
Peer-reviewedJ. P. A. Ioannidis · 2005 · PLoS Medicine 2(8):e124
Why Most Published Research Findings Are False
Bayes applied to the research enterprise: low priors + small effects + flexible analyses = unreliable positives. Limitation: a model-based argument whose inputs vary by field.

Reflect

There are no right answers to check against. These questions are for thinking on the page, and your answers go to your notebook.

Take a claim you currently believe about AI. Estimate your prior. What single piece of evidence would most change it, and by how much?

Not saved yetAll reflections →

Explain to a GTM colleague why a 90%-accurate lead-scoring model can still be wrong about most flagged leads.

Not saved yetAll reflections →

Where in your work do you use "entropy" or "signal-to-noise" loosely? What distribution would make it precise?

Not saved yetAll reflections →

What does a language model’s loss reward that you, as a writer, would not call good writing?

Not saved yetAll reflections →