gemma-2-9b
google/gemma-2-9bgemma-2-9b at Q4_K_M is exactly 5,761,057,664 bytes (5.37 GiB / 5.76 GB) — an effective 4.987 bits per weight, not the nominal 4. Its KV cache at 32K is 5.99 GiB, not the 10.50 GiB a flat formula predicts.
Shipped quantizations
| Quant | Size● | Exact bytes● | Effective bpw● | Tensors● | Publisher |
|---|---|---|---|---|---|
| Q2_K | 3.54 GiB | 3,805,397,888 | 3.294 | — | QuantFactory |
| Q3_K_S | 4.04 GiB | 4,337,664,896 | 3.755 | — | QuantFactory |
| Q3_K_M | 4.43 GiB | 4,761,781,120 | 4.122 | — | QuantFactory |
| Q3_K_L | 4.78 GiB | 5,132,452,736 | 4.443 | — | QuantFactory |
| Q4_0 | 5.07 GiB | 5,443,142,528 | 4.712 | — | QuantFactory |
| Q4_K_S | 5.10 GiB | 5,478,925,184 | 4.743 | — | QuantFactory |
| Q4_K_S | 5.10 GiB | 5,478,925,728 | 4.743 | — | INSAIT-Institute |
| Q4_K_M | 5.37 GiB | 5,761,057,664 | 4.987 | — | QuantFactory |
| Q4_K_M | 5.37 GiB | 5,761,058,208 | 4.987 | — | INSAIT-Institute |
| Q4_1 | 5.55 GiB | 5,963,367,296 | 5.162 | — | QuantFactory |
| Q5_K_S | 6.04 GiB | 6,483,592,064 | 5.612 | — | QuantFactory |
| Q5_0 | 6.04 GiB | 6,483,592,064 | 5.612 | — | QuantFactory |
| Q5_K_S | 6.04 GiB | 6,483,592,608 | 5.612 | — | INSAIT-Institute |
| Q5_K_M | 6.19 GiB | 6,647,366,528 | 5.754 | — | QuantFactory |
| Q5_1 | 6.52 GiB | 7,003,816,832 | 6.063 | — | QuantFactory |
| Q6_K | 7.07 GiB | 7,589,069,696 | 6.569 | — | QuantFactory |
| Q8_0 | 9.15 GiB | 9,827,148,672 | 8.507 | — | QuantFactory |
| Q8_0 | 9.15 GiB | 9,827,149,216 | 8.507 | — | INSAIT-Institute |
KV cache by context
| Context | KV cache (f16)● | Flat formula | Overstated by | Full / windowed / recurrent |
|---|---|---|---|---|
| 4,096 | 1.31 GiB | 1.31 GiB | — | 21 / 21 / 0 |
| 8,192 | 2.05 GiB | 2.63 GiB | 1.28× | 21 / 21 / 0 |
| 16,384 | 3.36 GiB | 5.25 GiB | 1.56× | 21 / 21 / 0 |
| 32,768 | 5.99 GiB | 10.50 GiB | 1.75× | 21 / 21 / 0 |
| 65,536 | 11.24 GiB | 21.00 GiB | 1.87× | 21 / 21 / 0 |
| 131,072 | 21.74 GiB | 42.00 GiB | 1.93× | 21 / 21 / 0 |
21 of 42 layers cache only a 4,096-token window rather than the full context, on a period of 2. Figures assume the default configuration; --swa-full disables the saving entirely.
Compare with
Will it run on your card?
Why other calculators give a different number
A parameters × bits ÷ 8 estimate puts Q4_K_M at roughly 4.84 GiB. The real file is 5.37 GiB, because a quantization is a mixture and some tensors are always kept at higher precision. The larger discrepancy is the cache: a flat formula gives 10.50 GiB at 32K context where the real figure is 5.99 GiB, because most of this model's layers cache a fixed window rather than the whole context.
Architecture
Questions people ask
- How much VRAM does gemma-2-9b need?
- Q4_K_M is exactly 5,761,057,664 bytes (5.37 GiB) in weights. Add the KV cache, which depends on your context length, plus roughly half a gigabyte of runtime overhead.
- How large is gemma-2-9b's KV cache?
- 5.99 GiB at 32K context with an f16 cache, computed per layer. Quantizing the cache to q8_0 roughly halves it, which is often the difference between a context length fitting and not.
- Which quantization of gemma-2-9b should I use?
- Q4_K_M is the usual default. Pick the largest quantization that fits your card at the context you actually need — the table above gives exact sizes for every one published.