How much worse is Q4? What quantization costs, and what nobody measured

The number you want does not exist

There is no public dataset of quality loss per quantization per model. Nobody maintains one. What circulates instead is a single scoreboard in llama.cpp's tools/perplexity/README.md, measured on LLaMA 3 8B at revision f364eb6f — a commit from 30 April 2024 — plus scattered figures in individual quantizers' README prose. Anyone quoting you a precise "Q4 loses 2.1% quality" for Qwen3 or GLM or Mistral Large is extrapolating from that one table, or inventing it.

Here is the table. Wikitext-2, CUDA, RTX 4090. The Q4-and-below rows use a 10M-token Wikitext importance matrix; Q5_K_M and above do not, because the README publishes no imatrix runs above Q4_K_S.

The f16 row is not zero because the reference logits are stored downcast to scaled 16-bit integers. That 0.00055 is the measurement floor, and Q8_0 sits at 2.5x it — which is another way of saying Q8_0 is indistinguishable from F16 by this method.

Each step down multiplies KL divergence: Q8 to Q6 is 4.0x, Q5 to Q4 is 2.6x, Q4 to Q3 is 3.0x, Q3 to Q2 is 3.9x. The multipliers compound off a base of nearly zero, so the absolute numbers are what matter. Everything from F16 down to Q4_K_M costs 2.4% perplexity in total; the single step from Q4_K_M to Q3_K_M costs another 5.6 points on top of that. Q8 to Q4 cheap, Q4 to Q3 expensive — that asymmetry is the most transferable finding here.

The average hides the damage

The detailed per-token statistics are published for the no-imatrix Q4_K_M build, which is slightly worse than the row above (PPL 6.4071, KLD 0.031273). Its mean change in the correct token's probability is -0.596%, and the median is -0.024%. That is why Q4 "feels fine": half the tokens are untouched to three decimal places.

The tails are where it lives. The 1st percentile is -19.567%, the 0.1st percentile -56.054%, the worst single token -98.699%. The README gives you the decision rule for reading these: symmetric percentiles mean quantization is adding noise, a heavier negative tail means the model is genuinely worse. Here the positive side reaches +12.084% at the 99th percentile and +27.084% at the 99.9th, so the negative tail is 1.6x and 2.1x larger respectively. Not noise. Real damage, concentrated in a few percent of tokens.

The consequence: at Q4_K_M, LLaMA 3 8B puts the top probability on a different token than F16 does 8.1% of the time (91.901% "same top p"). At Q6_K that is 4.0%, at Q8_0 2.3%. The failure mode is a dropped function argument or an invented flag inside an otherwise fluent paragraph, not generally mushier prose.

A quant label names a mixture, not a precision

llama.cpp promotes tensors it treats as fragile, via one predicate in src/llama-quant.cpp:

return i_layer < n_layers/8 || i_layer >= 7*n_layers/8 || (i_layer - n_layers/8)%3 == 2;

For Q4_K_M, layers satisfying it get attn_v and ffn_down in Q6_K instead of Q4_K, and output.weight is Q6_K unconditionally. Work it for Qwen3-8B — 36 layers, tie_word_embeddings: false: the first branch takes layers 0-3, the second takes 31-35, the third takes every third layer from 0, and the union is 18 distinct layers. That predicts 18 + 18 + 1 = 37 Q6_K tensors. The published file has exactly that: bartowski/Qwen_Qwen3-8B-GGUF's Qwen_Qwen3-8B-Q4_K_M.gguf is 399 tensors as 217 Q4_K + 145 F32 + 37 Q6_K, 5,027,784,224 bytes, 4.911 effective bpw against a nominal 4.5.

Nothing in the source explains why attn_v and ffn_down were chosen; the rule carries no comment. It is empirical. Quantize flat with --pure to delete the whole mixture and measure what those promotions buy on your own model.

The same quant hurts different models differently

Measured in the same run, same revision, same corpus:

Two models of nearly identical size, and Q4_K_M does 2.5 times more damage to the newer one. LLaMA 2 was pretrained on 2 trillion tokens, LLaMA 3 on over 15 trillion. Scaling Laws for Precision (Kumar et al.) predicts exactly this: post-training quantization degradation grows with pretraining tokens, to the point where more data becomes actively harmful once you quantize. Their fits come from 465 pretraining runs but top out at 1.7B parameters and 26B tokens, so applying the law at 8B is extrapolation. It is still the best available explanation, and 2026 models are trained on far more tokens per parameter. Expect them to tolerate Q4 worse than this table implies, not better.

Bigger model at lower quant, or smaller at higher?

Community consensus says take the bigger model down to about Q4. I agree with it, and it is consensus, not measurement.

The obstacle is structural: perplexity is not comparable across models with different tokenizers, which llama.cpp's own README states in its second paragraph. You cannot settle "70B at Q4 versus 32B at Q8" with the metric everyone quotes. The only support is within-family and weak. Llama 2 7B at Q4_K_M has a mean PPL ratio of 1.0142 against its own F16 and 1.1080 at Q2_K; the Llama 2 70B scoreboard shows 1.20% and 8.82% for the same two quants. Directionally right, but the 70B figures sit under an "Old Numbers" heading, carry no KL divergence, and predate the current methodology — and 1.20 versus 1.42 is not a mandate.

What is solid is the shape: below Q4 the bigger model's advantage erodes fast, because degradation accelerates faster than parameters compensate. Whether a 70B at Q2_K beats a 32B at Q5_K_M is unpublished.

Below roughly 4 bpw, prefer I-quants. At effectively identical file size, iq3_S (3.42 GiB) records 0.111278 KLD against q3_K_S (3.41 GiB) at 0.199821 — 44% less divergence for free. Same pattern at 2 bits: iq2_M is both smaller and better than q2_K, 2.74 GiB at 0.32599 versus 2.96 GiB at 0.33222. Above 4 bpw it inverts, and q4_K_S (4.37 GiB, 0.031951) beats iq4_NL (4.35 GiB, 0.035742).

What an imatrix actually does

An importance matrix is per-column mean squared activations, collected by running llama-imatrix over calibration text. It does not change the bit budget. It changes the error weighting inside the scale-fitting optimizer. In quantize_row_q4_K_impl in ggml/src/ggml-quants.c, without one each element is weighted by its own magnitude:

for (int l = 0; l < 32; ++l) weights[l] = av_x + fabsf(x[32*j + l]);

With one, the activation statistics enter as a multiplier:

for (int l = 0; l < 32; ++l) weights[l] = qw[l] * sqrtf(sigma2 + x[32*j + l]*x[32*j + l]);

The optimizer will then accept larger error on weights that inputs rarely excite in exchange for accuracy on the ones that matter. The benefit grows as bits shrink: on LLaMA 3 8B an imatrix cuts KL divergence by 10.0% at Q4_K_M, 17.2% at Q3_K_M and 25.4% at Q2_K. The common claim that it stops mattering at Q5 and above is an extrapolation off that trend — the README publishes no imatrix run above Q4_K_S, so nobody has measured it.

Calibration set size is oversold. The published sweep at Q2_K, in token order, gives KLD 0.337093 at 1k, 0.331393 at 10k, 0.331672 at 100k, 0.335308 at 1M and 0.332223 at 10M. Non-monotonic, and 10k beats 10M. The README's conclusion: "There seems to be no consistent improvement from using more Wikitext tokens for the importance matrix." Treat calibration-corpus size as a marketing claim.

Measuring it on your own model

Nobody will do this for you:

./scripts/get-wikitext-2.sh

# record F16 logits — 11 GiB for a LLaMA 2, 37 GiB for a LLaMA 3
./build/bin/llama-perplexity -m model-f16.gguf -f wikitext-2-raw/wiki.test.raw \
    --kl-divergence-base f16-logits.kld

# compare any quant against them
./build/bin/llama-perplexity -m model-Q4_K_M.gguf -f wikitext-2-raw/wiki.test.raw \
    --kl-divergence-base f16-logits.kld --kl-divergence

Read the Δp percentiles, not the mean, and apply the symmetry test above.

Then ignore Wikitext. It measures next-token prediction on encyclopedia prose. If you run the model for code, tool calls, or a language other than English, a good score there proves nothing about your workload. Collect 200 prompts from what you actually do, run them at two quants with temperature 0 and a fixed seed, and diff the outputs. It takes a weekend and it will tell you more than any table on the internet, including this one.