Chinchilla: The 20-Tokens-per-Parameter Rule and What Survived Its Replication

Core insights. Given a fixed compute budget $C \approx 6ND$ , loss is minimized by scaling model size $N$ and training tokens $D$ in equal proportion — roughly 20 tokens per parameter — not by growing $N$ and holding $D$ near 300B as Kaplan et al. (2020) implied. The empirical proof: Chinchilla (70B params, 1.4T tokens) beats Gopher (280B params, 300B tokens) at the same training compute, on essentially everything. Two caveats define how the result is used today: the analysis optimizes training compute only, so models meant to be served are deliberately trained far past 20 tok/param (“over-training”); and one of the paper’s three estimation approaches was later shown to be mis-fit, though correcting it strengthens the equal-scaling conclusion.

Method

Three independent estimates of the compute-optimal frontier $N_{opt}(C), D_{opt}(C)$ , using 400+ training runs from 70M to 16B parameters:

Fixed model sizes, varied token counts. For each $N$ , train across many horizons; take the loss envelope and read off the optimal $(N, D)$ per compute level.
IsoFLOP profiles. Fix nine compute budgets, vary $N$ at each; fit parabolas in $\log N$ and locate the minima.
Parametric loss fit. Fit a functional form to all runs:

\hat L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}

with reported values $E = 1.69$ , $A = 406.4$ , $B = 410.7$ , $\alpha = 0.34$ , $\beta = 0.28$ . Here $E$ is irreducible loss (entropy of text), the second term is the finite-capacity penalty, the third the finite-data penalty. Minimizing $\hat L$ subject to $6ND = C$ gives

N_{opt} \propto C^{a},\quad D_{opt} \propto C^{b},

with all three approaches landing near $a \approx b \approx 0.5$ — against Kaplan’s $a \approx 0.73,\ b \approx 0.27$ . The main causes of the disagreement, identified later: Kaplan’s runs did not tune the learning-rate schedule to the training horizon (cosine cycle length must match token budget) and fit in a small-model, few-token regime.

Claims & evidence

Claim	Evidence in paper	Verdict
$N$ and $D$ should scale equally with compute ( $a \approx b \approx 0.5$ )	Three convergent estimation approaches (§3.1–3.3)	verified — survived replication; the corrected Approach 3 fit agrees better with 0.5 than the original
~20 tokens/param is compute-optimal	Table 3 projections from the fits	partial — right order of magnitude; later analyses put the optimum anywhere in ~15–30 tok/param depending on data and tokenizer, and it drifts with data quality
Chinchilla 70B beats Gopher 280B at equal training FLOPs	Direct head-to-head: language modeling, MMLU, BIG-bench, QA (§4)	verified — the decisive experiment; consistent across nearly all evaluated tasks
Approach 3’s reported fit parameters	§3.3, Table A3	refuted as reported — Besiroglu et al. (2024) reconstructed the data from the paper’s figures and showed the reported $(A, B, \alpha, \beta, E)$ could not have come from the stated fitting procedure; their re-fit gives $a \approx 0.5$ with narrow confidence intervals, consistent with Approaches 1–2
MMLU 5-shot: Chinchilla 67.5% vs Gopher 60.0%	Table 6	verified as reported; treat 2022 MMLU protocols with the usual caution when comparing across papers

Benchmarks

Same training compute (~5.8 × 10²³ FLOPs), from the paper:

	Gopher	Chinchilla
Parameters	280B	70B
Training tokens	300B	1.4T
MMLU (5-shot avg)	60.0%	67.5%
Inference/fine-tune cost	1×	~0.25×

The last row is the strategically important one: the compute-optimal model is also 4× cheaper at inference — which is why the industry immediately moved past compute-optimal into over-training (LLaMA-1 7B: ~140 tok/param; later small models: thousands), trading extra training compute for a smaller served model. Chinchilla’s frontier tells you where training-compute efficiency peaks, not what to build.

Limitations & open questions

Single data distribution, single epoch. The fits assume unlimited fresh tokens from one MassiveText-like mixture. Data-constrained scaling (repeating epochs — Muennighoff et al. 2023) and data-quality effects change the constants materially.
The parametric form has no interaction term between $N$ and $D$ ; it cannot represent overfitting-like regimes and is known to be a local approximation.
Learning-rate-schedule sensitivity explains Kaplan-vs-Chinchilla, which means the “law” is partly a statement about tuned training practice, not a physical constant — expect drift as optimizers and schedules change.
Nothing here is about downstream capability per FLOP; loss-optimal and eval-optimal allocations can diverge, especially post-RLHF.

Reproduction notes

Marked partial: the full 400-run sweep is unreproducible outside a large lab, but the load-bearing pieces have been independently checked. Epoch AI’s replication (Besiroglu et al. 2024, arXiv 2404.10102) reconstructed ~240 runs from Figure 4’s SVG data and re-fit Approach 3 — finding the reported fit inconsistent, and the corrected fit consistent with equal scaling. The 20-tok/param heuristic has been re-derived in open settings (e.g., Cerebras-GPT, various isoFLOP studies at smaller scale), landing in the 15–30 range. Practical guidance that survives everything: match your LR schedule length to your token budget before trusting any scaling fit you run yourself.