Chinchilla: The 20-Tokens-per-Parameter Rule and What Survived Its Replication
Jordan Hoffmann et al., 2022 — NeurIPS 2022
Chinchilla showed that 2022-era LLMs were dramatically undertrained, replacing 'scale parameters' with 'scale parameters and tokens together.' A 2024 replication attempt found real problems in one of its three analyses — and the headline rule survived anyway.
repro: partial