How much VRAM does granite-4.0-350m need?

Q4_K_M is exactly 236,985,760 bytes (0.22 GiB) in weights. Add the KV cache, which depends on context length, plus roughly half a gigabyte of runtime overhead.

How large is granite-4.0-350m's KV cache?

0.88 GiB at 32K context with an f16 cache, computed per layer from the model's architecture. Quantizing the cache to q8_0 roughly halves it.

ibm-granite · text

granite-4.0-350m

ibm-granite/granite-4.0-350m

granite-4.0-350m at Q4_K_M is exactly 236,985,760 bytes (0.22 GiB / 0.24 GB) — an effective 5.380 bits per weight, not the nominal 4. Its KV cache at 32K is 0.88 GiB.

From the file· summed from 1 file(s)From the file· KV per layer

Parameters

352M

Architecture

granite

28 layers

Context

32,768

native (config.json)

License

apache-2.0

Shipped quantizations

● exact bytes, summed from published files

Quant	Size●	Exact bytes●	Effective bpw●	Tensors●	Publisher
UD-IQ3_XXS	0.16 GiB	172,142,368	3.908	—	unsloth
Q2_K	0.17 GiB	180,645,280	4.101	—	ibm-granite
Q3_K_S	0.18 GiB	195,325,344	4.434	—	ibm-granite
Q3_K_S	0.18 GiB	195,325,728	4.434	—	unsloth
Q3_K_M	0.19 GiB	208,326,048	4.730	—	ibm-granite
Q3_K_M	0.19 GiB	208,326,432	4.730	—	unsloth
Q3_K_L	0.20 GiB	219,925,920	4.993	—	ibm-granite
IQ4_XS	0.21 GiB	221,818,656	5.036	—	unsloth
Q4_0	0.21 GiB	228,470,176	5.187	—	ibm-granite
Q4_0	0.21 GiB	228,863,776	5.196	—	unsloth
Q4_K_S	0.21 GiB	229,387,680	5.208	—	ibm-granite
IQ4_NL	0.21 GiB	229,388,064	5.208	—	unsloth
Q4_K_S	0.21 GiB	229,388,064	5.208	—	unsloth
Q4_K_M	0.22 GiB	236,985,760	5.380	—	ibm-granite
Q4_K_M	0.22 GiB	236,986,144	5.380	—	unsloth
Q4_1	0.23 GiB	244,067,744	5.541	—	ibm-granite
Q4_1	0.23 GiB	244,068,128	5.541	—	unsloth
Q5_K_S	0.24 GiB	259,665,312	5.895	—	ibm-granite
Q5_0	0.24 GiB	259,665,312	5.895	—	ibm-granite
Q5_K_S	0.24 GiB	259,665,696	5.895	—	unsloth
Q5_K_M	0.25 GiB	264,052,128	5.995	—	ibm-granite
Q5_K_M	0.25 GiB	264,052,512	5.995	—	unsloth
Q5_1	0.26 GiB	275,262,880	6.249	—	ibm-granite
Q6_K	0.27 GiB	292,810,144	6.648	—	ibm-granite
Q6_K	0.27 GiB	292,810,528	6.648	—	unsloth
Q8_0	0.35 GiB	378,138,016	8.585	—	ibm-granite
Q8_0	0.35 GiB	378,138,400	8.585	—	unsloth
BF16	0.66 GiB	708,439,456	16.084	—	ibm-granite
BF16	0.66 GiB	708,439,552	16.084	—	unsloth

KV cache by context

computed per layer

Context	KV cache (f16)●	Flat formula	Overstated by	Full / windowed / recurrent
4,096	0.11 GiB	0.11 GiB	—	28 / 0 / 0
8,192	0.22 GiB	0.22 GiB	—	28 / 0 / 0
16,384	0.44 GiB	0.44 GiB	—	28 / 0 / 0
32,768	0.88 GiB	0.88 GiB	—	28 / 0 / 0
65,536	1.75 GiB	1.75 GiB	—	28 / 0 / 0
131,072	3.50 GiB	3.50 GiB	—	28 / 0 / 0

Compare with

same modality, comparable size

vs Llama-3.2-1B-Instruct vs ced-base vs embeddinggemma-300m vs Qwen3-0.6B vs Qwen2.5-0.5B-Instruct vs SmolLM2-135M-Instruct All alternatives →Will it fit on my card? →

Why other calculators give a different number

A parameters × bits ÷ 8 estimate puts Q4_K_M at roughly 0.18 GiB. The real file is 0.22 GiB, because a quantization is a mixture and some tensors are always kept at higher precision.

Architecture

from config.json

Layers

Attention heads

KV heads

Head dim

Hidden size

1024

Vocab

100,352

Sliding window

none

SWA period

—

MLA

Experts

Experts per token

use_sliding_window

—

Questions people ask

How much VRAM does granite-4.0-350m need?: Q4_K_M is exactly 236,985,760 bytes (0.22 GiB) in weights. Add the KV cache, which depends on your context length, plus roughly half a gigabyte of runtime overhead.
How large is granite-4.0-350m's KV cache?: 0.88 GiB at 32K context with an f16 cache, computed per layer. Quantizing the cache to q8_0 roughly halves it, which is often the difference between a context length fitting and not.
Which quantization of granite-4.0-350m should I use?: Q4_K_M is the usual default. Pick the largest quantization that fits your card at the context you actually need — the table above gives exact sizes for every one published.