Rosetta Stone · 30 of 30
Loss function (calligraphic L)
Naming convention: meaning depends on the field
In plain English
In ML, a single number measuring how wrong the model is. Lower is better, and training pushes it down. The curly is a typographic convention, not an operator, and it is overloaded: in physics is the Lagrangian (kinetic minus potential energy), and in statistics a similar script letter often denotes a likelihood, which you maximise. Check which one the paper means.
As code
// cross-entropy loss for one next-token prediction
const probs = { cat: 0.7, dog: 0.2, car: 0.1 };
const target = 'cat';
const loss = -Math.log(probs[target]); // ≈ 0.357 natsSee it
What it means, field by field
Training objective: cross-entropy, mean-squared error, RL losses; L(N) in scaling laws is test loss.
Lagrangian density ℒ = T − V; field theories are specified by writing down ℒ.
Likelihood function ℒ(θ | data), or a Lagrangian with multipliers in constrained optimisation.
Lower loss ≠ better on what you care about. A loss is a proxy chosen by someone.
In a research paper
Average, over every position in the text, of how surprised the model was by the token that actually came next.
Common misreadings
Reading a scaling-law L as the Lagrangian of physics, or the reverse.
Equating lower loss with "smarter". Loss measures next-token prediction on a particular distribution.
Mixing up likelihood (maximised) and loss (minimised). Negative log-likelihood is a loss.