Shannon entropy

EstablishedMathematicsAI / MLBiologyPhysics

\htmlClass{sjt-H}{H} = \htmlClass{sjt-minus}{-}\htmlClass{sjt-sum}{\sum_{i}} \htmlClass{sjt-p}{p_i}\,\htmlClass{sjt-log}{\log p_i}

Tap a symbol above, or choose one here:

H

Entropy

Meaning

Average uncertainty, or average information gained when you learn the outcome, measured in bits (log base 2) or nats (natural log).

Its job in this equation

The output: a single number summarising how unpredictable a distribution is.

Picture it

The average number of yes/no questions needed to guess the outcome.

Read it, progressively

1. Gist

How unpredictable is this? Zero when one outcome is certain; maximum when all outcomes are equally likely.

What it does not establish

Entropy is a property of a probability distribution, not of an object by itself. "The entropy of a DNA sequence" means the entropy of some distribution estimated from it, under some model. It ignores meaning: a random string has high entropy and no message. Estimating it from finite data is biased.

Play with it

Entropy & cross-entropy

p weight · A0.60p weight · C0.20p weight · G0.10p weight · T0.10q weight · A0.25q weight · C0.25q weight · G0.25q weight · T0.25

Entropy H(p)1.571 bits

Maximum possible (uniform)2.000 bits

Cross-entropy H(p,q)2.000 bits

KL(p‖q) = gap0.429 bits

$H(p)$ is the average surprise, $-\log p_i$ , when outcomes really follow $p$ . Cross-entropy $H(p,q)$ is the average surprise when reality is $p$ but you predict with $q$ . It is never below $H(p)$ , and the gap is the KL divergence. Language-model training minimises cross-entropy between real next tokens and the model’s predictions. The labels A, C, G, T are DNA bases, as in a sequence logo, but the arithmetic does not care what the outcomes are.

Loading interactive…

§ Module 4: Probability, Entropy, and Information Next equation: Cross-entropy (the language-model loss) →