EfficientQAT, low-bit training on a single GPU

Method

Where this sits

Our GPTQ note covers the cheap end: repair weights layer by layer with Hessian arithmetic, never train, four GPU-hours for 175B, dead at 2 bits. The ParetoQ note covers the expensive end: retrain everything on ~30B tokens, state of the art at 2 bits, a 16-GPU cluster for days. EfficientQAT (July 2024, so predating ParetoQ) is the deliberate middle, and its two phases map exactly onto the two things the expensive end buys: adaptation of weights to the grid, and coordination of errors across the whole network. It buys the first with block-local training and the second with a parameter subset small enough to fit anywhere.

Phase 1: Block-AP, or QAT one block at a time

Work through the transformer sequentially, one decoder block (attention + MLP, seven linear layers) at a time. For each block, capture its inputs on calibration data, then train all of it with gradient descent: the weights, the per-group scales s, and the zero points z, minimizing mean-squared error against the full-precision block’s outputs. When the block converges, freeze it, quantize it in place (block_ap.py:274), and move on. Memory never holds more than one block’s gradients and optimizer state, which is why a 70B model fits in 29.9 GB.

Three design choices carry the phase, and the paper ablates each (all numbers WikiText2 perplexity, Llama-2-7B w2g64):

1. Train everything, not clever subsets. The PTQ lineage this descends from (AdaRound, BRECQ, OmniQuant) trained only rounding flags or clipping parameters, constraining weights to move at most one grid step. EfficientQAT just unlocks it all: scales-only gets 6.09, weights-only 6.13, everything together 5.93 (vs 6.39 for plain rounding). The authors’ framing is that partial training was a regularizer against overfitting that also capped the solution space, and at 2 bits the cap costs more than the overfitting.
2. One block is the right granularity. Per-layer training gets 6.10, one block 5.93, two blocks 5.96, four 6.00. The sweet spot the BRECQ paper argued from Fisher-information structure shows up cleanly in LLM-scale measurements.
3. The loss is cross-stream, and this is the subtle one. The code keeps two activation streams flowing through the model (block_ap.py:75–157): the pristine FP stream, and the quantized stream that has passed through all previously-quantized blocks. Each block trains on quantized-stream inputs but regresses onto FP-stream targets, so it learns to fix its own quantization error and the accumulated drift inherited from upstream. The ablation says this asymmetric target is worth 0.07 perplexity over matching same-stream outputs. It costs double the activation storage, and I only understood the design after reading the dataset-update code, not the paper.

One more detail that connects to our ParetoQ discussion: the scales and the weights get separate learning rates on separate cosine schedules (quant 1e-4, weights 2e-5 at 2-bit; block_ap.py:190–210). The “let the scales move faster than the weights” idea I speculated about for ParetoQ is already standard practice here, implemented, amusingly, by stepping two empty AdamW optimizers purely to drive the schedulers.

Initialization is deliberately naive, and understanding why that’s safe here took me a moment. Weights start as the pristine pretrained values: each block is deep-copied from the FP model, so predecessors being quantized in place never contaminates the copy (int_linear_fake.py:25). Scales and zero points are per-group min–max, $s = (x_{max} - x_{min})/(2^N - 1)$, $z = \mathrm{round}(-x_{min}/s)$, both trainable from birth (quantizer.py:40–50); my check script confirms the group endpoints land on the grid’s first and last codes. At channel granularity this rule would be reckless, since one outlier coarsens thousands of weights, which is why the ParetoQ note agonizes over max-versus-MSE initialization. Here three things compound to make it nearly irrelevant: a group of 64 confines an outlier’s damage to its 63 neighbors; the trainable zero point lets the grid slide along a lopsided group rather than only stretch; and each block immediately receives ~4096 dense-supervision steps with scales learning at 1e-4, so a mediocre starting grid is repaired within a fraction of an epoch. The lesson I’d generalize: how much initialization matters is inversely proportional to how dense and immediate the correction signal is.

And a block, once trained, is not merely frozen but retired. Its weights are baked in place, it runs one final forward to advance the quant stream, and it moves to CPU, never revisited (block_ap.py:274–280). Since E2E-QP later trains only scales, every weight in the model gets exactly one training window in its life: its own block’s ~4096 iterations. All cross-block coordination afterward flows through the ~1.6% of parameters that are scales. That single design fact is simultaneously why a 70B fits on one GPU and where the accuracy ceiling against full QAT comes from.

Phase 2: E2E-QP, training without an STE

After Block-AP, the model is greedily good: each block matches its FP counterpart locally, but no block knows what the others need. Phase two fixes that with the cheapest possible end-to-end pass: pack the weights into real integers, freeze them and the zero points, and train only the step sizes on next-token cross-entropy over the target data.

Here is the property that made me stop and re-read. Because the integers are frozen, the forward pass contains no quantization operator, only dequantization: $\widehat{W} = (W_{int} - z)\cdot s$. That’s linear in $s$, so the gradient $\partial\widehat{W}/\partial s = W_{int} - z$ is exact. Every other QAT method in these notes trains through a rounding step and needs the straight-through estimator, a productive lie about a zero-derivative staircase. E2E-QP has nothing to lie about; it’s ordinary, honest gradient descent on a reparameterized model. I verified the exactness numerically (finite differences match the analytic gradient to 1e-10). Scales are ~1.6% of parameters at group size 64, so a 2-bit 70B trains end to end in 34.2 GB, and the whole phase doubles as continual pretraining (RedPajama) or instruction tuning (Alpaca) by just swapping the dataset.

The recipe in numbers

Both phases, defaults from the paper’s §4.1 and the repo’s example scripts (examples/block_ap/Llama-2-7b/w2g64.sh, examples/e2e_qp/.../w2g64-redpajama.sh):

	Block-AP	E2E-QP
Data	4096 RedPajama samples, 2048 tokens	4096 RedPajama samples, 4096 tokens
Trains	weights + scales + zero points, per block	scales only, whole model
Loss	MSE to FP block outputs (cross-stream)	next-token cross-entropy
LR	weights 2e-5, quant params 1e-4 (2-bit), cosine	2e-5 (2-bit), 1e-5 (3-bit)
Batch / epochs	2 / 2 per block, 64-sample val split	32 effective / 1
Gradient path	STE through rounding	exact, no rounding in graph

Cost on one A100-80GB, from the paper’s Table 8: 7B in 4.8 hours total, 13B in 8.5, 70B in 40.9 (26.6 Block-AP + 14.3 E2E-QP), peak memory 34.2 GB for the 70B 2-bit E2E phase. For calibration: ParetoQ’s 2-bit recipe consumes ~30B tokens on 16 GPUs; EfficientQAT’s whole pipeline sees ~0.03B unique tokens.

Paper vs. code

Repo: OpenGVLab/EfficientQAT at commit 39f37f3, official, actively maintained (the README notes ACL 2025 acceptance). The core is compact: quantize/quantizer.py (86 lines), quantize/block_ap.py (311), main_e2e_qp.py (534).

What I could match directly:

Paper	Code
Asymmetric uniform quantizer, Eq. 1–2	quantizer.py:58–74; my grid check reproduces 4 affine levels with 0.0 exactly representable
Min–max init of $s, z$	quantizer.py:40–50, endpoints land within scale/2 (verified)
Block-AP trains $W, s, z$	param groups in block_ap.py:186–210
E2E-QP trains step sizes only	main_e2e_qp.py:292 freezes all, :418 re-enables scales
Exact $\partial\widehat{W}/\partial s = W_{int} - z$	no quant op in the E2E path (int_linear_real); finite-difference verified
4096-sample calibration, batch 2, 2 epochs	defaults in main_block_ap.py:76–80

What the code does that the paper doesn’t say:

• The cross-stream loss (quantized inputs, FP targets) is implemented via the dual dataset machinery in block_ap.py and ablated in a table, but the method section never states it; the equations read as if inputs and targets came from one stream. It’s a real design ingredient, worth 0.07 ppl by their own measurement, living only in an ablation caption and the code.
• Zero points are nn.Parameters trained continuously through a round_ste in every forward (quantizer.py:60), then hard-rounded at pack time (block_ap.py:288). So “z is stored in low-bit format” is true only after training; during Block-AP it’s a float with STE gradients.
• The separate cosine schedules are driven by two empty AdamW optimizers created solely so CosineAnnealingLR has something to attach to (block_ap.py:195, 204). Harmless, funny, and invisible in the paper.
• Each block is cast to FP32 for training and back to FP16 after (block_ap.py:186, 273), so Block-AP’s real dtype story is mixed-precision per block, not the model’s native half precision.
• clamp_ste is defined twice, identically, back to back (quantizer.py:16–20). A copy-paste artifact; the second definition silently wins.
• An early-stopping mechanism on the per-block validation split exists but defaults off (main_block_ap.py:95); the published numbers run fixed epochs.

Claims & evidence

Zero-shot average below = five tasks (WinoGrande, PIQA, HellaSwag, ARC-e, ARC-c) via lm-eval-harness v0.4.2.

Claim	Evidence in paper	Verdict
2-bit Llama-2-70B within 3 points of FP16 (69.48 vs 72.41) in 41 GPU-hours	Tables 1, 8; released checkpoints on HuggingFace	verified as reported; I did not re-run
Beats all uniform-quantization baselines at 2/3-bit (GPTQ, AWQ, OmniQ, AutoRound, DB-LLM)	Table 1 across five models	verified per the table, e.g. +3.2 over DB-LLM on Llama-2-7B w2g64
Competitive with vector quantization	Their own Table 1: QuIP# wins at 2-bit on all three Llama-2 sizes (e.g. 70.91 vs 69.48 at 70B)	partial — the paper says this plainly and argues deployability; the accuracy race at 2-bit goes to VQ
Beats Q-PEFT methods (QLoRA, QA-LoRA, PEQA, IR-QLoRA) for low-bit instruction tuning	MMLU comparison, Fig. 3; E2E-QP matches 16-bit LoRA (44.23 vs 44.15) at 3.3 effective bits	verified as reported
First method to train all parameters block-wise	§3.2 priority claim	unverified — plausible for LLM quantization; priority claims are hard to audit
Block-AP full training beats partial-parameter variants	Ablation: 5.93 vs 6.09/6.13	verified as a measurement, on one model/bit setting

Benchmarks

Five-task zero-shot average, higher is better, from the paper’s Table 1 (w2g64 rows plus context):

Method	Bits	L2-7B	L2-13B	L2-70B	L3-8B
FP16	16	64.86	67.81	72.41	68.58
OmniQ	2 (g128)	46.98	53.56	54.87	52.66
DB-LLM	2 (g64)	56.93	61.61	68.01	51.74
AQLM	2 (vector)	57.61	62.22	69.85	n/a
EfficientQAT	2 (g64)	60.14	64.48	69.48	60.76
QuIP#	2 (vector)	60.61	64.44	70.91	n/a

Two honest readings. Within uniform quantization, EfficientQAT was clearly the 2-bit frontier when published, and the margins over OmniQuant-era PTQ are enormous on Llama-3, the family that quantizes worst. Against everything, its 2-bit crown lasted about seven months: QuIP#‘s codebooks edge it in this very table, and ParetoQ later blew past it (71.2 on Llama-3-8B) by spending the compute this paper’s whole point is to avoid.

And a cross-paper discrepancy that turned out to be instructive once I chased it down. ParetoQ’s baseline table credits “EfficientQAT w2g128” on Llama-3-8B with 65.5, while EfficientQAT’s own paper reports 59.37 for that exact setting (the released checkpoint’s HuggingFace card says 59.36). Same checkpoint, same five tasks, same g128 config, and the six-point gap is entirely acc versus acc_norm. EfficientQAT’s per-task appendix table says so in its caption (“acc is reported, not acc_norm”); ParetoQ’s table reports acc_norm, which on multiple-choice tasks like HellaSwag and ARC-challenge runs far above raw accuracy (ParetoQ’s own full-precision HellaSwag is 79.5, which is an acc_norm figure; the plain acc is around 60). The clean tell is that the same offset shows up on the full-precision model, where quantization plays no part: this Llama-3-8B scores 68.58 under acc (EfficientQAT) and 74.6 under acc_norm (ParetoQ), a 6.0-point gap, essentially the same 6.1 points that separate 59.37 from 65.5. I checked that arithmetic in the verification script. Neither number is wrong, they are non-comparable across the boundary, and that is the lesson worth keeping: a cross-paper table can shift more from the choice of accuracy metric than from the methods it means to rank. ParetoQ’s own internal comparisons are all acc_norm, so those are fair; it is only EfficientQAT’s self-reported 59.37 (acc) that must not be read against them.

Limitations & open questions

The 2-bit accuracy crown is gone, and the interesting question is the exchange rate. On the fair comparison (both acc_norm, from ParetoQ’s table) ParetoQ beats EfficientQAT’s Llama-3-8B 2-bit result by 5.7 points, 71.2 to 65.5, for roughly a thousand times the training compute. Nobody has mapped the frontier between 41 GPU-hours and 30B tokens; the obvious hybrid, Block-AP as the initialization for a real QAT run, is exactly the “compensated init” experiment discussed in the ParetoQ note, and this repo is the natural starting point for it.

Group-wise bookkeeping cuts both ways: g64 costs ~0.25 extra effective bits over channel-wise, and cross-paper tables rarely normalize for it.

The evaluation surface is soft. The six-point cross-paper discrepancy above turned out to be a metric mismatch (acc vs acc_norm), not noise, but that is if anything the more unsettling finding: the choice of accuracy metric moved this cell by more than most of the method gaps in the same table, and the paper’s own care in pinning lm-eval v0.4.2 shows how much the harness details matter.

Weight-only by default. The paper cites follow-up work (PrefixQuant) for weight-activation settings, but the shipped recipe quantizes weights only; activation outliers, the harder half of the problem at deployment time, are out of scope.

Everything is Llama-family. No Mistral/Qwen/MoE results in the main tables (the repo has since added Mistral-Large examples), and instruction-tuning evidence is Llama-1-era MMLU protocol, following QA-LoRA for comparability.

Error propagation through the greedy chain is treated but never measured. The cross-stream loss exists to absorb upstream drift, and its ablation caption says as much, yet the paper contains no plot of reconstruction error versus block index, and neither the repo nor its issue tracker has one either. The code logs exactly this number for every block of every run (block_ap.py:262); to my knowledge nobody has published the sequence, so “how does error actually accumulate over 32 or 80 blocks” is an open question that costs three GPU-hours to answer. The granularity ablation hints the answer may be unintuitive: jointly training 2 or 4 blocks does worse than 1 (5.96 and 6.00 vs 5.93), so at 4096 calibration samples the binding constraint appears to be overfitting rather than propagation, which quietly undercuts the premise of cross-block methods like CBQ at this data budget.

Reproduction notes

Marked partial, meaning: on 2026-07-13 I read the quantizer, the Block-AP loop, and the E2E-QP trainer end to end at commit 39f37f3 and matched them against the paper’s equations and ablations; that’s the Paper-vs-code section. I reimplemented the quantizer in NumPy and verified the grid structure, the min–max initialization, the exactness of the E2E-QP scale gradient, and the round-STE value identity; the script is verification/efficientqat/checks.py, all PASS. Benchmark numbers were transcribed from the LaTeX table source.

While checking the issue tracker I found one piece of third-party evidence worth recording: a user independently re-ran Block-AP on Llama-2-7B and landed within 0.11 perplexity of the officially released weights (issue #13, 7.76 vs 7.65 on WikiText2). That’s a small but genuine independent reproduction of phase 1 from public artifacts, which strengthens the “verified as reported” verdicts above.

What I did not do: run either training phase (the 7B pipeline is ~5 GPU-hours and would be the cheapest full reproduction on this site so far, which makes it a good candidate), or evaluate the released HuggingFace checkpoints. The one confirming eval left for the acc-vs-acc_norm story is cheap: re-run the released ChenMnZ/Llama-3-8b-EfficientQAT-w2g128 checkpoint and check it gives ~59.4 under acc and ~65.5 under acc_norm; the per-task arithmetic already makes this near-certain, but I have not run it. The single most interesting cheap experiment remains the per-block loss trajectory from the Limitations section: one 7B Block-AP run plus a log parser produces the error-versus-depth curve for block-wise QAT, a figure that, as far as I can tell, does not exist in the literature.