Best local AI models for 4GB VRAM
Ranked by what actually fits at 32K context, computed from real file bytes.
A 4GB card gives you about 3.72 GiB to work with after driver overhead. 33 indexed models fit at 32K context — the largest being Qwen3.5-9B-Base at 9.7B parameters in IQ3_XS.
From the file· fit from summed bytesFrom the file· KV per layer
Fits in 4GB at 32K context
largest quantization that fits, per model
| Model | Modality | Best quant | Params○ | Total◐ | Headroom◐ |
|---|---|---|---|---|---|
| Qwen3.5-4B | vision + language | IQ2_M | 4.7B | 3.63 GiB | 0.09 GiB |
| Qwen3.5-0.8B | vision + language | BF16 | 873M | 2.60 GiB | 1.12 GiB |
| gemma-4-E2B-it | vision + language | IQ3_XXS | 5.1B | 3.51 GiB | 0.21 GiB |
| Qwen3.5-2B | vision + language | Q8_0 | 2.3B | 3.11 GiB | 0.61 GiB |
| LFM2.5-VL-1.6B | vision + language | BF16 | 1.6B | 3.37 GiB | 0.35 GiB |
| MiniCPM-V-4.6 | vision + language | BF16 | 1.3B | 2.57 GiB | 1.15 GiB |
| LocateAnything-3B | vision + language | Q3_K_M | 3.8B | 3.54 GiB | 0.18 GiB |
| OvisOCR2 | vision + language | BF16 | 853M | 2.57 GiB | 1.15 GiB |
| gemma-3n-E2B-it | vision + language | Q3_K_M | 5.4B | 3.54 GiB | 0.18 GiB |
| Qwen2-VL-2B-Instruct | vision + language | Q8_0 | 2.2B | 3.21 GiB | 0.51 GiB |
| medgemma-4b-it | vision + language | IQ4_XS | 4.3B | 3.72 GiB | 0.00 GiB |
| Qwen3.5-0.8B-Base | vision + language | BF16 | 873M | 2.60 GiB | 1.12 GiB |
| LFM2-VL-1.6B | vision + language | BF16 | 1.6B | 3.37 GiB | 0.35 GiB |
| LFM2.5-VL-450M | vision + language | F32 | 449M | 2.48 GiB | 1.24 GiB |
| Qwen3.5-2B-Base | vision + language | Q8_0 | 2.3B | 3.05 GiB | 0.67 GiB |
| Qwen3.5-9B-Base | vision + language | IQ3_XS | 9.7B | 3.29 GiB | 0.43 GiB |
| gemma-4-E2B-it-qat-q4_0-unquantized-heretic | vision + language | I1-IQ3_XXS | 5.1B | 3.51 GiB | 0.21 GiB |
| LFM2-VL-450M | vision + language | BF16 | 451M | 1.82 GiB | 1.90 GiB |
| medgemma-1.5-4b-it | vision + language | Q3_K_L | 4.3B | 3.69 GiB | 0.03 GiB |
| LFM2-VL-3B | vision + language | Q6_K | 3.0B | 3.27 GiB | 0.45 GiB |
| Gemma-3-4B-VL-it-Gemini-Pro-Heretic-Uncensored-Thinking | vision + language | Q3_K_L | 4.3B | 3.69 GiB | 0.03 GiB |
| Qwen3.5-0.8B-heretic | vision + language | Q8_0 | 853M | 2.17 GiB | 1.55 GiB |
| Qwen3.5-4B-heretic | vision + language | BF16 | 4.5B | 3.01 GiB | 0.71 GiB |
| NuExtract3 | vision + language | Q2_K | 4.5B | 3.60 GiB | 0.12 GiB |
| Qwen3.5-2B-heretic | vision + language | Q8_0 | 2.2B | 3.56 GiB | 0.16 GiB |
| InternVL3_5-8B | vision + language | IQ2_M | 8.5B | 3.69 GiB | 0.03 GiB |
| Nanonets-OCR-s | vision + language | Q4_K_S | 3.8B | 3.65 GiB | 0.07 GiB |
| gemma-3-4b-it-abliterated | vision + language | Q3_K_L | 4.3B | 3.69 GiB | 0.03 GiB |
| MiniCPM-V-4.6-Thinking | vision + language | F16 | 1.3B | 2.57 GiB | 1.15 GiB |
| AfriqueQwen3.5-4B | vision + language | I1-IQ2_M | 5.2B | 3.69 GiB | 0.03 GiB |
| AREX-Turbo | vision + language | IQ2_M | 4.5B | 3.51 GiB | 0.21 GiB |
| Fara1.5-4B | vision + language | IQ2_M | 4.5B | 3.51 GiB | 0.21 GiB |
| Qwen3.5-2B-enko | vision + language | Q8_0 | 2.1B | 2.83 GiB | 0.89 GiB |
This page models a generic 4GB accelerator, so it answers what fits rather than how fast it runs. For tokens per second you need a specific card — pick one from hardware, where bandwidth is known.