Topic
7 notes · all notes
concept note
Weight quantization is most of this site; the harder half of deployment is activations, where a few feature channels run tens of times larger than the rest and wreck any shared grid. Three methods define the lineage: LLM.int8 keeps the outliers in fp16, SmoothQuant migrates them into the weights, QuaRot rotates them away. I audited all three repos and verified the mechanism of each in NumPy.
Tianyi Chen et al., 2026 — arXiv preprint (ICML 2026 format)
Quantization-aware training fakes the gradient of the rounding step; almost everyone fakes it with the straight-through estimator (pretend round's derivative is 1). StableQAT replaces that constant with a smooth surrogate derived by rotating the rounding staircase into a triangle wave and Fourier-expanding it, with STE as the zero-amplitude special case. I read the Microsoft repo and verified the surrogate's math: the STE limit, the amplitude conditioning boundary, and the bounded-variance theorem.
Zechun Liu et al., 2024 — ICLR 2025
QuIP proved random rotations make models quantizable. SpinQuant asks the obvious next question: why random? It parameterizes the rotations, descends the quantized network's own loss on the orthogonal manifold for 100 steps, and the answer is decisive, especially at 70B where random rotations can collapse. This is also the method Meta actually ships on-device. I audited the repo, verified the manifold math numerically, and reproduced learned-beats-random in miniature.
Mengzhao Chen et al., 2024 — ACL 2025
EfficientQAT is the middle point between GPTQ's four-hour repair job and ParetoQ's thirty-billion-token retrain: a 2-bit Llama-2-70B on one A100 in 41 hours, within 3 points of full precision. I read the paper and audited the repo. The two-phase design is cleaner than I expected, one phase quietly needs no straight-through estimator at all, and the numbers around it have a genuine cross-paper mystery.
Jerry Chee et al., 2023 — NeurIPS 2023
QuIP is where LLM quantization got its theory: a proof that GPTQ was secretly optimal within its class, a definition of what makes matrices hard to quantize, and the two-sided random rotation that fixes it. Every rotation-based method since descends from this paper. I audited the Cornell repo, reproduced the central theorem numerically to exact equality, and traced which pieces the successors kept.
Zechun Liu et al., 2025 — NeurIPS 2025
I went looking for the current state of the art in 2- and 3-bit quantization-aware training, with the requirement that the evidence be auditable. ParetoQ won: a 2-bit Llama-3 8B within 3.4 points of full precision, an official repo whose core is one file, and three findings about training that matter beyond quantization. Includes what the code does that the paper's equation gets slightly wrong.
Elias Frantar et al., 2022 — ICLR 2023
GPTQ shrinks a trained model's weights from 16 bits to 3 or 4 bits while barely changing what the model computes. I read the paper and its 171-line reference implementation side by side, and wrote this up so that it needs no prior background in quantization. The math is built up from small numerical examples.