Topic
training
3 notes · all notes
Jul 18, 2026 quantization / training / llm
Tianyi Chen et al., 2026 — arXiv preprint (ICML 2026 format)
Quantization-aware training fakes the gradient of the rounding step; almost everyone fakes it with the straight-through estimator (pretend round's derivative is 1). StableQAT replaces that constant with a smooth surrogate derived by rotating the rounding staircase into a triangle wave and Fourier-expanding it, with STE as the zero-amplitude special case. I read the Microsoft repo and verified the surrogate's math: the STE limit, the amplitude conditioning boundary, and the bounded-variance theorem.
repro: partial 4 verified2 partial
Jul 13, 2026 quantization / training / llm
Mengzhao Chen et al., 2024 — ACL 2025
EfficientQAT is the middle point between GPTQ's four-hour repair job and ParetoQ's thirty-billion-token retrain: a 2-bit Llama-2-70B on one A100 in 41 hours, within 3 points of full precision. I read the paper and audited the repo. The two-phase design is cleaner than I expected, one phase quietly needs no straight-through estimator at all, and the numbers around it have a genuine cross-paper mystery.
repro: partial 4 verified1 partial1 unverified
Jul 12, 2026 quantization / training / llm
Zechun Liu et al., 2025 — NeurIPS 2025
I went looking for the current state of the art in 2- and 3-bit quantization-aware training, with the requirement that the evidence be auditable. ParetoQ won: a 2-bit Llama-3 8B within 3.4 points of full precision, an official repo whose core is one file, and three findings about training that matter beyond quantization. Includes what the code does that the paper's equation gets slightly wrong.
repro: partial 3 verified2 partial1 unverified