WHY THE LINESARE STRAIGHT

Grow one resource while holding the other fixed. The result depends on their hierarchy.

D counts training examples; P counts student features in the solvable linear model. Underparameterized means D ≫ P; overparameterized means P ≫ D. When the growing resource is already abundant, the model approaches a smooth limiting prediction. When that resource is scarce, it still resolves new structure.

§1.1–1.2 · §2 · Fig. 1a

Width is not parameter count. In the deep-network scaling described here, w ∝ √P: a width loss-gap exponent of one corresponds to P^−1/2. In linear models, variance-limited feature scaling is P⁻¹. The resolution panels in Fig. 1a also measure width; they do not supply αP directly.

§1.1 · §2.1.2 · §2.3.1

The network’s prediction fluctuates around a limit. Smooth loss turns shrinking variance into a shrinking loss gap.

Concentration means repeated training runs give increasingly similar predictions as a resource grows. Dataset fluctuations have variance of order 1/D; width fluctuations have variance of order 1/w. Theorem 1 transfers that order to expected loss when the centered moments and loss satisfy its conditions.

§2.1 · Theorem 1 · App. B–C

App. C.3 · scope distinction from the language-model fits

A manifold is a space described locally by a smaller number of independent coordinates. Its intrinsic dimension is the number of directions the model needs to resolve.

An image has many pixel coordinates, but those coordinates need not vary independently. High input dimension alone says little about intrinsic manifold dimension. Under the paper’s compact-manifold assumptions, a fresh point gets closer to its nearest training point as D grows.

§2.2.1–2.2.2 · Theorems 2–3 · App. D

If both target and student change at bounded rates, matching a training target constrains error nearby. Such a rate bound is called Lipschitz continuity. Theorems 2–3 turn it into upper bounds on test loss, for sampled data or a model that interpolates on a parameter-controlled set of points.

Theorems 2–3

An upper bound says how bad the error can be. An estimate says how the error usually behaves. The authors make that extra step explicitly.

Expand loss near the closest training point. If the first surviving local term has order n, raising the spacing to that power gives n/d in the loss exponent. A sufficiently accurate piecewise-linear fit can push the first nonzero term to fourth order. That motivates αD ≈ 4/d.

§2.2.3 · bounds as estimates

When teacher–student input dimension is controlled, 4/αD tracks it. That experiment supports the estimate in a clean setting. For convolutional networks and Wide ResNets on standard datasets, the relationship is less clear. The clean test does not make intrinsic dimension an observed constant for arbitrary real data.

§3.1 · Fig. 1b

Fix the features. Learn their weights. The network becomes a problem that can be solved exactly.

A feature is a fixed function of an input; a random-feature model samples such functions before training and learns only their linear combination. A teacher creates targets from a feature pool. The student receives a P-dimensional projection of that pool and D examples, then reaches the global optimum of mean squared error—the average squared prediction error—from zero initial weights.

This construction has a connection to wide networks. A Gaussian process describes random functions through Gaussian-distributed values; suitable infinite-width networks admit that description. A neural tangent kernel describes similarities through parameter gradients; under the corresponding wide-network training limit it stays fixed. Those limits motivate the model without making it a description of every finite network.

§2.3 · Eq. 1 · App. E

A kernel measures similarity between inputs. Its eigenvalues rank the strength of independent modes the student could learn.

The corresponding feature covariance measures second moments between features. Both carry the same spectral information. The eigenvalue spectrum is the ordered list of mode strengths; spectral decay describes how quickly they fall with rank. A fast-falling spectrum leaves less important structure unresolved.

§2.3.2–2.3.4 · Eqs. 4–6

A C^t kernel has t continuous derivatives. Smoothness limits how heavy its eigenvalue tail can be. On an intrinsic d-dimensional space, the bound involves t/d; if the spectrum saturates that bound, it gives a dimension-dependent exponent.

§2.3.4 · smooth kernels on the d-torus

Smooth functions on low-dimensional manifolds have spectra that fall fast. Fast-falling spectra mean steep scaling. That is the bridge from geometry to exponent—with the bound-saturation assumption kept in view.

§2.3.4 · saturation assumption

Project onto random features, or onto random training points. The exact linear loss expressions exchange one projection for the other.

With a shared power-law spectrum, the resolution-limited loss exponent for features at abundant data equals the exponent for data at abundant features: αP = αD = αK. Fig. 2b tests the relationship with pooled MNIST and random ReLU features. Appendix F also compares loss curves using frozen EfficientNet-B5 features under low and tuned regularization.

Double descent means error can rise near interpolation and fall again as a resource grows. Sample-wise double descent is one realization of this duality, rather than a separate general explanation of every scaling curve.

§2.4 · S37–S38 · Fig. 2b · Fig. S6

The empirical sweep separates what stays nearly fixed from what moves.

Tests include MNIST, FashionMNIST, CIFAR-10, CIFAR-100 and SVHN; fully connected, convolutional and Wide ResNet architectures; ReLU and Erf activations; mean squared error and cross-entropy; and changes to stochastic-gradient batches. These settings support the taxonomy, with variance-limited exponents near one and resolution-limited exponents that depend on data and model.

§3.1–3.3 · App. A, F · Figs. 1a, S1–S3, S7

Superclassing merges fine labels into broader categories. In these tests, that changes loss level while leaving the data exponent similar.

Gaussian noise changes the inputs themselves, and the exponent falls as noise grows. The instrument displays the reported superclass band and the verified noise endpoints, without inventing intermediate measurements.

§3.3 · Fig. 3

The authors interpret the contrast as evidence that networks learn input-manifold structure independently of the precise classification task, resembling unsupervised learning. The measured contrast is evidence; that mechanism remains an interpretation.

§4 · interpretation of Fig. 3

§3.3 · App. A.8 · Fig. S5

Nothing in this section is established by the paper. The paper studies vision datasets, linear and random-feature models, and wide networks. Language models are not its subject.

The comparison asks what its mechanism might suggest elsewhere. It supplies hypotheses to test, with assumptions that can fail.

Context Jamming extension · paper scope: §1–4

For a further pointer: Sharma and Kaplan, Scaling laws from the data manifold dimension, JMLR (2022). The Chinchilla chapter provides the allocation comparison; Dyer’s profile provides context for the emergence discussion. Neither establishes the analogies above.

Related work · reference [10], not this paper

Explaining Neural Scaling Laws

Bahri, Y., Dyer, E., Kaplan, J., Lee, J., & Sharma, U. Explaining Neural Scaling Laws. arXiv:2102.06701 [cs.LG] · v1: 12 February 2021 · v2: 29 April 2024. Published in PNAS 121(27), e2311878121 (2024). DOI: 10.1073/pnas.2311878121.

Yasaman Bahri · Ethan Dyer · Jared Kaplan · Jaehoon Lee · Utkarsh Sharma

Google DeepMind, Mountain View (Bahri, Dyer, Lee); Physics and Astronomy, Johns Hopkins University (Kaplan, Sharma).

All authors contributed to all aspects of the work. Part of Sharma’s work was completed during a Google internship. Kaplan and Sharma were supported in part by Open Philanthropy.

The paper combines theoretical derivations in fixed-feature models with empirical vision experiments. This explainer reads v2; no underlying training-run data were recovered or invented.

Primary record · attached arXiv v2; journal fields verified against arXiv

Curves are deterministic calculations or clearly labelled conceptual reconstructions. The spectral calculator sums a finite toy spectrum; its integral approximation extends to infinity. Eq. 6 is shown only up to a constant because its printed prefactor conflicts with the integral. The regime table states the actual D/P hierarchy, and the smoothness control keeps the bound’s direction explicit.

Visualization method