Three ways around activation outliers

The problem: a few channels are enormous

The weight-only notes on this site quantize $W$ and leave the activations $X$ in 16 bits. That is the easy half. The moment you want int8 or int4 activations you meet a distribution that does not cooperate: in a trained transformer past a few billion parameters, a small number of feature channels (columns of $X$) hold values ten to a hundred times larger than the typical entry, and they sit in the same few dimensions for every token. A per-tensor grid stretched to cover a value of 70 gives a step of about $70/127$ for int8, so an ordinary activation near 1 lands on the first or second tick and everything that made it distinct is gone.

This is the activation-side version of the range problem the white paper solved for weights with cross-layer equalization, and its own most quoted sentence is a footnote admitting that BERT needed “a few quantized activations kept in 16 bits” to hold accuracy, eighteen months before anyone named the cause. LLM.int8’s measurement made it concrete: the systematic outliers emerge in a phase transition around 6.7B parameters, where the fraction of layers they touch jumps from roughly two-thirds to all of them (paper says, arXiv 2208.07339); they are about 0.1% of feature dimensions, and zeroing them degrades validation perplexity by 600 to 1000 percent. They are rare, structured, and load-bearing, which is exactly why all three methods below treat the channels differently from the bulk.

Keep them: LLM.int8

The first answer is not to quantize the outliers at all. LLM.int8 splits each activation-times-weight product in two: the handful of feature columns whose magnitude exceeds a threshold are pulled out and multiplied in fp16, the rest are quantized to int8 and multiplied there, and the two partial results are summed. Because the loud columns are gone from the int8 half, that half’s per-row absmax is small and the ordinary values keep their resolution. I checked the whole point of the decomposition in NumPy: on an activation matrix with two outlier columns, plain per-token int8 lands an error 13 times larger than the mixed path that reserves those two columns for fp16.

In the reference code (bitsandbytes, commit f2233a6) the decomposition is backends/default/ops.py:64-100: gather the outlier columns A[:, outlier_cols] in fp16, int8-matmul the bulk, and output.addmm(subA, subB) to add the fp16 contribution back. The int8 result is denormalized by the outer product of the row and column absmax scales divided by $127^2$ (default/ops.py:57), which is the paper’s vector-wise dequant written as one constant; $1/127^2 = 6.2\times10^{-5}$, and I verified that is the number.

Two things the paper does not tell you and the code does. The famous threshold of 6.0, the magnitude above which a feature is declared an outlier, is not in bitsandbytes: the library default is threshold=0.0 (nn/modules.py:1056, autograd/_functions.py:72), i.e. no decomposition at all unless a caller asks for it. The 6.0 lives one layer up, in HuggingFace Transformers’ BitsAndBytesConfig(llm_int8_threshold=6.0), which passes it down. And the “fp16” outlier weights are not the original fp16 numbers: the inference path dequantizes the int8 weight columns back to fp16 on the fly (default/ops.py:81-87, with a TODO noting the exact-fp16 path is not taken), so only the outlier activations are truly full precision. The paper’s clean decomposition and the shipped one differ by exactly that.

The price of this method is structural. The fp16 columns make the matmul irregular, and at batch size 1, where a decode step is memory-bound anyway, the extra bookkeeping means LLM.int8 is about keeping accuracy at int8, not about speed. It is the safe answer, and the one every later method tried to avoid paying for.

Migrate them: SmoothQuant

The second answer accepts that the activations are hard and the weights are easy, and moves difficulty from one to the other. For each input channel $j$, SmoothQuant computes a smoothing scale from the channel’s activation range and weight range,

$$s_j = \frac{\max_t |X_{tj}|^{\alpha}}{\max_o |W_{jo}|^{1-\alpha}} ,$$

then divides the activations by $s$ and multiplies the weights by it: $X \mapsto X\,\mathrm{diag}(s)^{-1}$, $W \mapsto \mathrm{diag}(s)\,W$. The product $XW$ is untouched, so the network computes the same function, and the scale folds into the previous layer’s LayerNorm weight and bias so there is no extra runtime operation (smooth.py:34-45, commit c61476d). I verified both halves in NumPy: the migration preserves the layer output to $7\times10^{-15}$, and at the default $\alpha=0.5$ each channel’s new activation and weight maxima both land on the geometric mean $\sqrt{\max|X_j|\cdot\max|W_j|}$ of the originals, the outlier channel in my toy dropping from a max of 120 to 5 while the weights it feeds rise from 0.4 to 5. The knob $\alpha$ slides the split: near 0 the activations stay hard, near 1 the weights do, and the paper’s sweet spot is 0.4 to 0.6, rising to 0.75 for GLM-130B where ~30% of channels are outliers (paper says, arXiv 2211.10438).

The code again says more than the equation. The paper’s $\alpha=0.5$ default is not what the released eval uses: examples/ppl_eval.sh sets $\alpha=0.85$ for Llama-2-7B and 13B, 0.9 for 70B, 0.8 for Mistral and Mixtral, values that live only in a shell script and are far from the headline 0.5. There are two separate clamps at $10^{-5}$ guarding the weight scale and the final scale (smooth.py:32,36), and only the norm-to-first-linear boundaries are smoothed, so the second matmul in each block is left alone. The dramatic evidence for the method is the failure it prevents: naive per-tensor W8A8 on OPT-175B collapses to near-chance accuracy with a five-figure perplexity, and SmoothQuant brings the same W8A8 setting back to near-lossless (paper says). What it does not do is remove the outliers; it balances them, which is enough for W8A8 and runs out of room well before W4A4.

Rotate them: QuaRot

The third answer makes the outliers disappear into a change of basis. Multiply the hidden state by a random orthogonal matrix and a spike in one coordinate is smeared across all of them; the heavy tail becomes near-Gaussian, which is exactly the incoherence QuIP manufactures for weights, now applied to activations. QuaRot uses a fixed randomized Hadamard as that rotation. I checked the smearing: a vector with one coordinate at 30 has a max-to-RMS ratio of 10.8, and a normalized Hadamard drops it to 1.6.

The reason rotating the residual stream is affordable is SliceGPT’s computational invariance, and it turns on a detail. QuaRot first folds every RMSNorm’s scale vector into the following linear and replaces the norm with a scale-free one (rotation_utils.py:45-85, commit 5008669). A scale-free RMSNorm commutes with any orthogonal $Q$, since normalizing by a norm that rotation preserves is the same before or after the rotation; I verified $\mathrm{RMSNorm}(xQ) = \mathrm{RMSNorm}(x)\,Q$ to $4\times10^{-15}$. Once that holds, one hidden-size $Q$ can be fused into the read-in of every stream-facing projection as $WQ$ and the read-out as $Q^\top W$, and it cancels across each residual block for free (rotation_utils.py:229-250). Where no fold-through exists, after RoPE and after the SwiGLU product, QuaRot applies a Hadamard online instead (quant_utils.py:220-242), the same four-site map SpinQuant uses. With outliers gone, weights go to 4 bits (GPTQ by default, gptq_utils.py), activations and the KV cache to 4 bits per token, and Llama-2 W4A4 with a 4-bit cache lands within a few points of full precision: 7B perplexity 6.10 against 5.47, 70B 3.79 against 3.32, where SmoothQuant at the same budget is at 83 (paper says, arXiv 2404.00456).

QuaRot sits precisely between the other two rotation notes. It is the fixed version of SpinQuant, which learns $Q$ by Cayley descent on calibration data; the rotation sites are the same, the difference is whether you optimize them. And it is the computational-invariance application of QuIP: QuIP calibrates two incoherence matrices per weight and stays weight-only, while QuaRot uses one shared random Hadamard and reaches activations and the cache. The code’s honesty tax is dimensional. The Hadamard sizes are hardcoded from Sloane’s tables factored as $K\cdot 2^m$ (hadamard_utils.py), there is no padding, and unsupported sizes assert-fail; the KV rotation needs a power-of-two head dimension. The online Hadamards are also real inference compute, which is why the fast tensor-core kernels came in follow-up work rather than this repo.

The three trades

Laid side by side, the methods are three points on one axis: how hard you work to make the activation distribution quantizable, against how much of the outlier problem you actually remove.

Method	Move	Reaches	Runtime cost	What it leaves
LLM.int8	keep outliers in fp16	W8A8, lossless	irregular fp16 path, no small-batch speedup	the outliers, untouched
SmoothQuant	migrate to weights	W8A8	none (folded)	a per-model $\alpha$; harder weights
QuaRot	rotate away	W4A4 + 4-bit KV	online Hadamards; nice dimensions only	rotations to run or fuse

The progression is the same lesson the weight side keeps teaching. LLM.int8 routes around the bad distribution, SmoothQuant rebalances it, and QuaRot changes the basis so the distribution is no longer bad, which is the QuIP and SpinQuant move made to serve activations. Every one of these is ultimately a statement that you should quantize in a coordinate system where the numbers are boring, and the history here is just successive discoveries of how cheaply you can get into one.

What I verified

The three mechanisms are one seeded NumPy script, verification/activation-outliers/checks.py, all PASS: SmoothQuant’s migration preserving the layer function and equalizing to the geometric mean at $\alpha=0.5$; LLM.int8’s decomposition beating plain int8 by 13× on outlier-laden input, with the $1/127^2$ dequant scale; and QuaRot’s Hadamard cutting an outlier’s max-to-RMS ratio while a scale-free RMSNorm commutes with the rotation. The code references are from reading the three reference repos at the commits cited above (bitsandbytes f2233a6, SmoothQuant c61476d, QuaRot 5008669); the current bitsandbytes has refactored the 2022 CUDA path into PyTorch custom ops, so its file layout is newer than the paper’s. The accuracy and perplexity numbers are the papers’, transcribed and labeled paper-says; I re-ran no model, so this note is a mechanism audit, not a reproduction of any table.