Best local AI models for 12GB VRAM

Ranked by what actually fits at 32K context, computed from real file bytes.

A 12GB card gives you about 11.16 GiB to work with after driver overhead. 1238 indexed models fit at 32K context — the largest being Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled at 36.0B parameters in I1-IQ2_XXS.

From the file· fit from summed bytesFrom the file· KV per layer
4GB6GB8GB10GB12GB16GB20GB24GB32GB48GB64GB96GB128GB
alltext generation 1238

Fits in 12GB at 32K context

largest quantization that fits, per model
ModelModalityBest quantParams○Total◐Headroom◐
gemma-4-12B-it-qat-q4_0-unquantizedtext generationQ4_012.0B9.81 GiB1.35 GiB
gemma-4-E4B-ittext generationQ8_08.0B8.95 GiB2.21 GiB
Qwen3-30B-A3B-Thinking-2507MoEtext generationIQ2_XXS30.5B10.84 GiB0.32 GiB
Qwen3-4Btext generationQ8_04.0B9.30 GiB1.86 GiB
Qwen3-8Btext generationQ5_K_L8.2B11.14 GiB0.02 GiB
Laguna-XS-2.1MoEtext generationIQ2_XXS33.4B10.93 GiB0.23 GiB
Llama-3.2-1B-Instructtext generationF161.2B4.11 GiB7.05 GiB
KAT-Coder-V2.5-DevMoEtext generationIQ2_XXS34.7B10.54 GiB0.62 GiB
llama-3-youko-8btext generationQ5_K_M8.0B10.18 GiB0.98 GiB
Llama-3.1-8B-Instructtext generationQ6_K8.0B10.98 GiB0.18 GiB
ced-basetext generationF3286M1.16 GiB10.00 GiB
Qwen2.5-7B-Instructtext generationQ8_07.6B10.15 GiB1.01 GiB
UI-TARS-1.5-7Btext generationQ8_08.3B10.15 GiB1.01 GiB
gemma-3-1b-ittext generationF161000M2.81 GiB8.35 GiB
gemma-4-E2B-it-qat-q4_0-unquantizedtext generationBF165.1B9.83 GiB1.33 GiB
Qwen3-1.7Btext generationBF162.0B8.08 GiB3.08 GiB
GLM-4.7-FlashMoEtext generationUD-IQ1_S31.2B11.07 GiB0.09 GiB
embeddinggemma-300mtext generationF32303M2.05 GiB9.11 GiB
Llama-3.2-3B-Instructtext generationF163.2B10.30 GiB0.86 GiB
Qwen3-0.6Btext generationBF16752M5.68 GiB5.48 GiB
Qwen3-14Btext generationUD-IQ2_M14.8B10.91 GiB0.25 GiB
Ornith-1.0-35BMoEtext generationIQ2_XXS34.7B10.54 GiB0.62 GiB
Wan2.1-T2V-1.3Btext generationQ5_01.4B8.58 GiB2.58 GiB
LFM2.5-1.2B-Instructtext generationBF161.2B3.37 GiB7.79 GiB
Qwen2.5-Coder-7B-Instructtext generationQ4_07.6B10.86 GiB0.30 GiB
Qwen2.5-1.5B-Instructtext generationF161.5B4.56 GiB6.60 GiB
Jan-v3-4B-base-instructtext generationQ8_04.4B9.68 GiB1.48 GiB
gemma-3-4b-ittext generationBF164.3B8.85 GiB2.31 GiB
MiniCPM5-1B-Claude-Opus-Fable5-Thinkingtext generationF161.1B3.55 GiB7.61 GiB
Ornith-1.0-9Btext generationQ8_09.2B10.95 GiB0.21 GiB
gemma-4-26B-A4B-it-ultra-uncensored-hereticMoEtext generationI1-IQ2_XXS25.8B10.99 GiB0.17 GiB
Qwen3-4B-Instruct-2507text generationQ8_04.0B9.30 GiB1.86 GiB
granite-4.1-3btext generationBF163.4B9.65 GiB1.51 GiB
Qwen2.5-3B-Instructtext generationF163.1B7.69 GiB3.47 GiB
gemma-2-2b-ittext generationQ8_02.6B5.26 GiB5.90 GiB
Qwen3-30B-A3B-Instruct-2507MoEtext generationIQ2_XXS30.5B10.84 GiB0.32 GiB
MiniCPM5-1B-Claude-Opus-Fable5-V2-Thinkingtext generationF161.1B3.55 GiB7.61 GiB
Qwen2.5-0.5B-Instructtext generationF16494M2.08 GiB9.08 GiB
DeepSeek-R1-0528-Qwen3-8Btext generationQ5_K_L8.2B11.14 GiB0.02 GiB
Qwen3-VL-8B-Instructtext generationQ5_K_M8.8B10.78 GiB0.38 GiB
Sugoi-14B-Ultra-HFtext generationI1-IQ2_XXS14.8B10.86 GiB0.30 GiB
gemma-4-12b-heretic-abliteratedtext generationI1-Q5_K_S12.0B11.08 GiB0.08 GiB
TinyLlama-1.1B-Chat-v1.0text generationF161.1B3.53 GiB7.63 GiB
Mistral-Nemo-Instruct-2407text generationQ3_K_S12.2B11.00 GiB0.16 GiB
Phi-4-mini-instructtext generationQ8_03.8B8.61 GiB2.55 GiB
SmolLM2-135M-Instructtext generationF16135M1.72 GiB9.44 GiB
gemma-3-27b-ittext generationUD-IQ1_M27.4B10.50 GiB0.66 GiB
FastContext-1.0-4B-SFTtext generationQ8_04.0B9.30 GiB1.86 GiB
DeepSeek-R1-Distill-Qwen-7Btext generationQ8_07.6B10.15 GiB1.01 GiB
GLM-4.6V-Flashtext generationQ6_K_L10.3B10.07 GiB1.09 GiB
Qwen2.5-Coder-3B-Instructtext generationF163.1B7.69 GiB3.47 GiB
Mistral-7B-Instruct-v0.3text generationQ6_K7.2B10.38 GiB0.78 GiB
Meta-Llama-3-8B-Instructtext generationQ6_K8.0B10.98 GiB0.18 GiB
Ternary-Bonsai-8B-unpackedtext generationQ4_K_M8.2B10.01 GiB1.15 GiB
Qwen2.5-Coder-1.5B-Instructtext generationF161.5B4.56 GiB6.60 GiB
Ministral-3-14B-Reasoning-2512text generationUD-IQ3_XXS13.9B10.97 GiB0.19 GiB
gemma-3-270m-ittext generationF16268M1.38 GiB9.78 GiB
umt5-xxltext generationQ8_05.7B6.48 GiB4.68 GiB
DeepSeek-R1-Distill-Qwen-1.5Btext generationF321.8B8.30 GiB2.86 GiB
LFM2.5-8B-A1BMoEtext generationQ8_08.5B9.56 GiB1.60 GiB
○Spec sheet◐Predictedwhat these mean

This page models a generic 12GB accelerator, so it answers what fits rather than how fast it runs. For tokens per second you need a specific card — pick one from hardware, where bandwidth is known.