An illustrative neural scaling law

Empirical regularityAI / MLMathematicsComplexity

\htmlClass{sjt-L}{L(N)} = \htmlClass{sjt-Linf}{L_{\infty}} + \htmlClass{sjt-A}{A}\,\htmlClass{sjt-N}{N}^{\htmlClass{sjt-alpha}{-\alpha}}

Tap a symbol above, or choose one here:

L(N)

Test loss as a function of NRosetta entry →

Meaning

Average cross-entropy (next-token surprise) on held-out text, for a model with N parameters.

Its job in this equation

The output. Lower is better, and the equation predicts how it falls as N grows.

Picture it

A function call: loss(N).

Read it, progressively

1. Gist

Bigger models do better, with diminishing returns that follow a predictable curve down toward a floor.

What it does not establish

This equation does not say: that loss tracks the capabilities you care about; that the trend continues beyond the measured range; that the same α holds for other data, architectures or modalities; or that $L_\infty$ is a fundamental constant. It is presented here as an illustrative form. Kaplan et al.’s own fits use slightly different parameterisations.

Play with it

Power-law explorer

L(N) = 1.70 + 12.0\,N^{-0.080}

α, exponent0.080A, scale12.0L∞, irreducible floor1.70Probe model size N10^9.0

L at probe3.987

L at 10× params3.602

10× buys (reducible part)×0.832

10× buys (absolute)−0.385

Illustrative constants, not a fit to real models. On log–log axes the reducible part, $AN^{-\alpha}$ , is a straight line with slope −α. The full curve bends toward the floor $L_\infty$ , the loss that no amount of scale removes in this model. On linear axes nearly all of the change happens at the far left, which is why scaling papers plot on log axes.

Loading interactive…

§ Module 1: Learning to Read Equations Next equation: Newton’s second law →