Cross-entropy (the language-model loss)

EstablishedAI / MLMathematics

\htmlClass{sjt-H}{H(p,q)} = -\htmlClass{sjt-sum}{\sum_{i}} \htmlClass{sjt-p}{p_i}\,\htmlClass{sjt-logq}{\log q_i}

Tap a symbol above, or choose one here:

H(p,q)

Cross-entropyRosetta entry →

Meaning

Average surprise when outcomes really follow p but you predict with q.

Its job in this equation

The output, and the quantity language-model training minimises.

Picture it

Your average score when the real world deals the cards (p) but you bet using your beliefs (q).

Read it, progressively

1. Gist

How surprised your model is, on average, by what really happens.

What it does not establish

Cross-entropy measures probabilistic prediction of the training distribution, not truthfulness, helpfulness or reasoning. Two models with equal loss can differ in behaviour. It is also not symmetric: $H(p,q) \ne H(q,p)$ .

Play with it

Entropy & cross-entropy

p weight · A0.60p weight · C0.20p weight · G0.10p weight · T0.10q weight · A0.25q weight · C0.25q weight · G0.25q weight · T0.25

Entropy H(p)1.571 bits

Maximum possible (uniform)2.000 bits

Cross-entropy H(p,q)2.000 bits

KL(p‖q) = gap0.429 bits

$H(p)$ is the average surprise, $-\log p_i$ , when outcomes really follow $p$ . Cross-entropy $H(p,q)$ is the average surprise when reality is $p$ but you predict with $q$ . It is never below $H(p)$ , and the gap is the KL divergence. Language-model training minimises cross-entropy between real next tokens and the model’s predictions. The labels A, C, G, T are DNA bases, as in a sequence logo, but the arithmetic does not care what the outcomes are.

Loading interactive…

§ Module 4: Probability, Entropy, and Information Next equation: Gradient descent update →