Best local AI models for 6GB VRAM
Ranked by what actually fits at 32K context, computed from real file bytes.
A 6GB card gives you about 5.58 GiB to work with after driver overhead. 502 indexed models fit at 32K context — the largest being granite-20b-code-instruct-8k at 20.1B parameters in IQ1_M.
From the file· fit from summed bytesFrom the file· KV per layer
Fits in 6GB at 32K context
largest quantization that fits, per model
| Model | Modality | Best quant | Params○ | Total◐ | Headroom◐ |
|---|---|---|---|---|---|
| gemma-4-E4B-it | text generation | Q3_K_S | 8.0B | 4.92 GiB | 0.66 GiB |
| Llama-3.2-1B-Instruct | text generation | F16 | 1.2B | 4.11 GiB | 1.47 GiB |
| ced-base | text generation | F32 | 86M | 1.16 GiB | 4.42 GiB |
| Qwen2.5-7B-Instruct | text generation | Q2_K | 7.6B | 5.41 GiB | 0.17 GiB |
| UI-TARS-1.5-7B | text generation | Q2_K | 8.3B | 5.41 GiB | 0.17 GiB |
| gemma-3-1b-it | text generation | F16 | 1000M | 2.81 GiB | 2.77 GiB |
| gemma-4-E2B-it-qat-q4_0-unquantized | text generation | Q4_K | 5.1B | 4.22 GiB | 1.36 GiB |
| Qwen3-1.7B | text generation | Q4_1 | 2.0B | 5.54 GiB | 0.04 GiB |
| embeddinggemma-300m | text generation | F32 | 303M | 2.05 GiB | 3.53 GiB |
| Llama-3.2-3B-Instruct | text generation | Q2_K | 3.2B | 5.58 GiB | 0.00 GiB |
| Qwen3-0.6B | text generation | Q8_0 | 752M | 5.02 GiB | 0.56 GiB |
| Wan2.1-T2V-1.3B | text generation | Q5_K_M | 1.4B | 3.44 GiB | 2.14 GiB |
| LFM2.5-1.2B-Instruct | text generation | BF16 | 1.2B | 3.37 GiB | 2.21 GiB |
| Qwen2.5-Coder-7B-Instruct | text generation | Q2_K | 7.6B | 5.41 GiB | 0.17 GiB |
| Qwen2.5-1.5B-Instruct | text generation | F16 | 1.5B | 4.56 GiB | 1.02 GiB |
| gemma-3-4b-it | text generation | Q8_0 | 4.3B | 5.46 GiB | 0.12 GiB |
| MiniCPM5-1B-Claude-Opus-Fable5-Thinking | text generation | F16 | 1.1B | 3.55 GiB | 2.03 GiB |
| Ornith-1.0-9B | text generation | IQ2_M | 9.2B | 5.44 GiB | 0.14 GiB |
| granite-4.1-3b | text generation | Q5_K_M | 3.4B | 5.57 GiB | 0.01 GiB |
| Qwen2.5-3B-Instruct | text generation | Q8_0 | 3.1B | 5.30 GiB | 0.28 GiB |
| gemma-2-2b-it | text generation | Q8_0 | 2.6B | 5.26 GiB | 0.32 GiB |
| MiniCPM5-1B-Claude-Opus-Fable5-V2-Thinking | text generation | F16 | 1.1B | 3.55 GiB | 2.03 GiB |
| Qwen2.5-0.5B-Instruct | text generation | F16 | 494M | 2.08 GiB | 3.50 GiB |
| TinyLlama-1.1B-Chat-v1.0 | text generation | F16 | 1.1B | 3.53 GiB | 2.05 GiB |
| SmolLM2-135M-Instruct | text generation | F16 | 135M | 1.72 GiB | 3.86 GiB |
| DeepSeek-R1-Distill-Qwen-7B | text generation | Q2_K | 7.6B | 5.41 GiB | 0.17 GiB |
| GLM-4.6V-Flash | text generation | UD-IQ2_XXS | 10.3B | 5.36 GiB | 0.22 GiB |
| Qwen2.5-Coder-3B-Instruct | text generation | Q8_0 | 3.1B | 5.30 GiB | 0.28 GiB |
| Qwen2.5-Coder-1.5B-Instruct | text generation | F16 | 1.5B | 4.56 GiB | 1.02 GiB |
| gemma-3-270m-it | text generation | F16 | 268M | 1.38 GiB | 4.20 GiB |
| umt5-xxl | text generation | Q6_K | 5.7B | 5.20 GiB | 0.38 GiB |
| DeepSeek-R1-Distill-Qwen-1.5B | text generation | BF16 | 1.8B | 4.99 GiB | 0.59 GiB |
| LFM2.5-8B-A1BMoE | text generation | UD-IQ4_NL | 8.5B | 5.23 GiB | 0.35 GiB |
| SmolVLM-500M-Instruct | text generation | F16 | 507M | 2.78 GiB | 2.80 GiB |
| tinygemma3_cifar | text generation | Q8_0 | 39M | 0.93 GiB | 4.65 GiB |
| Ministral-3-3B-Reasoning-2512 | text generation | Q2_K | 4.3B | 5.42 GiB | 0.16 GiB |
| Qwen3-0.6B-Base | text generation | F16 | 596M | 5.39 GiB | 0.19 GiB |
| Qwen2-7B-Instruct | text generation | IQ3_XXS | 7.6B | 5.50 GiB | 0.08 GiB |
| VibeThinker-3B | text generation | Q8_0 | 3.1B | 5.00 GiB | 0.58 GiB |
| Yi-1.5-6B-Chat | text generation | Q3_K_S | 6.1B | 5.35 GiB | 0.23 GiB |
| Ministral-3-3B-Instruct-2512-BF16 | text generation | IQ3_XS | 4.3B | 5.53 GiB | 0.05 GiB |
| Qwen2.5-VL-3B-Instruct | text generation | Q8_0 | 3.8B | 5.00 GiB | 0.58 GiB |
| t5-v1_1-xxl | text generation | Q2_K | 4.8B | 5.56 GiB | 0.02 GiB |
| Qwen3.6-27B-DFlash | text generation | F16 | 1.7B | 4.26 GiB | 1.32 GiB |
| Qwen3-Reranker-0.6B | text generation | F16 | 596M | 5.39 GiB | 0.19 GiB |
| LFM2.5-230M | text generation | BF16 | 230M | 1.57 GiB | 4.01 GiB |
| LFM2.5-350M | text generation | BF16 | 354M | 1.82 GiB | 3.76 GiB |
| Nanbeige4.2-3B | text generation | IQ3_M | 4.2B | 5.51 GiB | 0.07 GiB |
| MiniCPM5-1B | text generation | F16 | 1.1B | 3.55 GiB | 2.03 GiB |
| TinyLlama-1.1B-Chat-v0.3 | text generation | Q8_0 | 1.1B | 2.57 GiB | 3.01 GiB |
| Qwen3.5-9B-Claude-4.6-Opus-Deckard-V4.2-Uncensored-Heretic-Thinking | text generation | I1-IQ3_XXS | 9.4B | 5.50 GiB | 0.08 GiB |
| Qwen2.5-VL-7B-Instruct-abliterated | text generation | I1-IQ3_XXS | 8.3B | 5.50 GiB | 0.08 GiB |
| Agents-A1-4B | text generation | Q4_K_M | 4.5B | 4.33 GiB | 1.25 GiB |
| SmolLM3-3B | text generation | Q6_K_L | 3.1B | 5.48 GiB | 0.10 GiB |
| gemma-2-2b-it-abliterated | text generation | Q8_0 | 2.6B | 5.26 GiB | 0.32 GiB |
| Qwen3.5-9B-Claude-4.6-HighIQ-THINKING-HERETIC-UNCENSORED | text generation | I1-IQ3_XXS | 9.4B | 5.37 GiB | 0.21 GiB |
| Ministral-3-3B-Instruct-2512 | text generation | Q2_K | 3.8B | 5.42 GiB | 0.16 GiB |
| Huihui-Qwythos-9B-Claude-Mythos-5-1M-abliterated | text generation | I1-Q2_K | 9.7B | 5.48 GiB | 0.10 GiB |
| LFM2.5-Audio-1.5B | text generation | F16 | 1.5B | 3.52 GiB | 2.06 GiB |
| SmolLM2-360M-Instruct | text generation | F16 | 362M | 2.69 GiB | 2.89 GiB |
This page models a generic 6GB accelerator, so it answers what fits rather than how fast it runs. For tokens per second you need a specific card — pick one from hardware, where bandwidth is known.