A technical notebook on machine learning systems
Model Foundations turns papers, source code, and benchmark tables into structured technical notes — methods worked through equation by equation, claims checked against their evidence, results placed beside their baselines, and open questions stated plainly.
concept note
A '2-bit' model almost never stores two bits per weight. Group scales, zero points, and vector-quantization codebooks all add overhead, and papers count that overhead differently, so the bit column in a benchmark table is itself a result. I worked out the arithmetic behind the numbers this site quotes (2.0, 2.12, 2.25, 2.27) and where they stop being comparable.
concept note
Weight quantization is most of this site; the harder half of deployment is activations, where a few feature channels run tens of times larger than the rest and wreck any shared grid. Three methods define the lineage: LLM.int8 keeps the outliers in fp16, SmoothQuant migrates them into the weights, QuaRot rotates them away. I audited all three repos and verified the mechanism of each in NumPy.
concept note
A uniform quantizer has one knob, the step size. LSQ's contribution was to learn it by gradient descent on the task loss, with a step-size gradient that is sensitive to how close each value sits to a rounding boundary, plus a scale factor that keeps its updates balanced against the weights'. SEQ is ParetoQ's low-bit descendant: the same learned step size, a better-shaped grid. I read the LSQ paper and its LaTeX, audited the quantizer code that four other notes here depend on, and verified the gradient and both grids.
Tianyi Chen et al., 2026 — arXiv preprint (ICML 2026 format)
Quantization-aware training fakes the gradient of the rounding step; almost everyone fakes it with the straight-through estimator (pretend round's derivative is 1). StableQAT replaces that constant with a smooth surrogate derived by rotating the rounding staircase into a triangle wave and Fourier-expanding it, with STE as the zero-amplitude special case. I read the Microsoft repo and verified the surrogate's math: the STE limit, the amplitude conditioning boundary, and the bounded-variance theorem.
Zechun Liu et al., 2024 — ICLR 2025
QuIP proved random rotations make models quantizable. SpinQuant asks the obvious next question: why random? It parameterizes the rotations, descends the quantized network's own loss on the orthogonal manifold for 100 steps, and the answer is decisive, especially at 70B where random rotations can collapse. This is also the method Meta actually ships on-device. I audited the repo, verified the manifold math numerically, and reproduced learned-beats-random in miniature.
Mengzhao Chen et al., 2024 — ACL 2025
EfficientQAT is the middle point between GPTQ's four-hour repair job and ParetoQ's thirty-billion-token retrain: a 2-bit Llama-2-70B on one A100 in 41 hours, within 3 points of full precision. I read the paper and audited the repo. The two-phase design is cleaner than I expected, one phase quietly needs no straight-through estimator at all, and the numbers around it have a genuine cross-paper mystery.
What actually matters in the work, in a few sentences.
The mechanism from first principles, with equations checked by hand.
The algorithm matched line by line against the reference repo.
Each headline claim traced to the experiment behind it.
Results next to their baselines, with the setup that produced them.
What the paper does not show, and open questions.
What was actually run or read, stated precisely.