Best local AI models for 8GB VRAM

Ranked by what actually fits at 32K context, computed from real file bytes.

A 8GB card gives you about 7.44 GiB to work with after driver overhead. 806 indexed models fit at 32K context — the largest being Darwin-36B-Opus at 34.7B parameters in IQ2_M.

From the file· fit from summed bytesFrom the file· KV per layer
4GB6GB8GB10GB12GB16GB20GB24GB32GB48GB64GB96GB128GB
alltext generation 806

Fits in 8GB at 32K context

largest quantization that fits, per model
ModelModalityBest quantParams○Total◐Headroom◐
gemma-4-E4B-ittext generationQ5_K_M8.0B6.69 GiB0.75 GiB
Qwen3-4Btext generationIQ4_XS4.0B7.43 GiB0.01 GiB
Llama-3.2-1B-Instructtext generationF161.2B4.11 GiB3.33 GiB
Llama-3.1-8B-Instructtext generationUD-IQ2_XXS8.0B7.17 GiB0.27 GiB
ced-basetext generationF3286M1.16 GiB6.28 GiB
Qwen2.5-7B-Instructtext generationQ4_K_L7.6B7.34 GiB0.10 GiB
UI-TARS-1.5-7Btext generationQ4_K_M8.3B6.97 GiB0.47 GiB
gemma-3-1b-ittext generationF161000M2.81 GiB4.63 GiB
gemma-4-E2B-it-qat-q4_0-unquantizedtext generationQ4_K5.1B4.22 GiB3.22 GiB
Qwen3-1.7Btext generationQ8_02.0B6.31 GiB1.13 GiB
embeddinggemma-300mtext generationF32303M2.05 GiB5.39 GiB
Llama-3.2-3B-Instructtext generationQ6_K_L3.2B6.86 GiB0.58 GiB
Qwen3-0.6Btext generationBF16752M5.68 GiB1.76 GiB
Wan2.1-T2V-1.3Btext generationQ4_01.4B7.32 GiB0.12 GiB
LFM2.5-1.2B-Instructtext generationBF161.2B3.37 GiB4.07 GiB
Qwen2.5-Coder-7B-Instructtext generationQ4_K_L7.6B7.34 GiB0.10 GiB
Qwen2.5-1.5B-Instructtext generationF161.5B4.56 GiB2.88 GiB
Jan-v3-4B-base-instructtext generationQ3_K_M4.4B7.40 GiB0.04 GiB
gemma-3-4b-ittext generationQ8_04.3B5.46 GiB1.98 GiB
MiniCPM5-1B-Claude-Opus-Fable5-Thinkingtext generationF161.1B3.55 GiB3.89 GiB
Ornith-1.0-9Btext generationQ4_19.2B7.37 GiB0.07 GiB
Qwen3-4B-Instruct-2507text generationIQ4_XS4.0B7.43 GiB0.01 GiB
granite-4.1-3btext generationQ8_03.4B6.67 GiB0.77 GiB
Qwen2.5-3B-Instructtext generationQ8_03.1B5.30 GiB2.14 GiB
gemma-2-2b-ittext generationQ8_02.6B5.26 GiB2.18 GiB
MiniCPM5-1B-Claude-Opus-Fable5-V2-Thinkingtext generationF161.1B3.55 GiB3.89 GiB
Qwen2.5-0.5B-Instructtext generationF16494M2.08 GiB5.36 GiB
gemma-4-12b-heretic-abliteratedtext generationI1-IQ2_M12.0B7.39 GiB0.05 GiB
TinyLlama-1.1B-Chat-v1.0text generationF161.1B3.53 GiB3.91 GiB
Phi-4-mini-instructtext generationQ5_K_S3.8B7.35 GiB0.09 GiB
SmolLM2-135M-Instructtext generationF16135M1.72 GiB5.72 GiB
FastContext-1.0-4B-SFTtext generationI1-IQ4_XS4.0B7.43 GiB0.01 GiB
DeepSeek-R1-Distill-Qwen-7Btext generationQ4_K_L7.6B7.34 GiB0.10 GiB
GLM-4.6V-Flashtext generationQ4_010.3B7.19 GiB0.25 GiB
Qwen2.5-Coder-3B-Instructtext generationQ8_03.1B5.30 GiB2.14 GiB
Mistral-7B-Instruct-v0.3text generationQ2_K7.2B7.37 GiB0.07 GiB
Meta-Llama-3-8B-Instructtext generationIQ2_XS8.0B7.27 GiB0.17 GiB
Ternary-Bonsai-8B-unpackedtext generationQ2_08.2B7.36 GiB0.08 GiB
Qwen2.5-Coder-1.5B-Instructtext generationF161.5B4.56 GiB2.88 GiB
gemma-3-270m-ittext generationF16268M1.38 GiB6.06 GiB
umt5-xxltext generationQ8_05.7B6.48 GiB0.96 GiB
DeepSeek-R1-Distill-Qwen-1.5Btext generationBF161.8B4.99 GiB2.45 GiB
LFM2.5-8B-A1BMoEtext generationUD-Q5_K_M8.5B7.10 GiB0.34 GiB
SmolVLM-500M-Instructtext generationF16507M2.78 GiB4.66 GiB
tinygemma3_cifartext generationQ8_039M0.93 GiB6.51 GiB
Qwen3.6-14B-A3B-FableVibesMoEtext generationQ2_K13.8B6.39 GiB1.05 GiB
Llama3.3-8B-Instruct-Thinking-Heretic-Uncensored-Claude-4.5-Opus-High-Reasoningtext generationI1-IQ2_S8.0B7.41 GiB0.03 GiB
Qwen3-4B-Thinking-2507text generationIQ4_XS4.0B7.43 GiB0.01 GiB
Ministral-3-3B-Reasoning-2512text generationQ6_K4.3B6.69 GiB0.75 GiB
Qwen3-0.6B-Basetext generationF16596M5.39 GiB2.05 GiB
Qwen2-7B-Instructtext generationQ4_K_M7.6B6.97 GiB0.47 GiB
VibeThinker-3Btext generationQ8_03.1B5.00 GiB2.44 GiB
Qwythos-9B-Claude-Mythos-5-1M-uncensored-heretictext generationQ4_K_S9.4B7.37 GiB0.07 GiB
Yi-Coder-9B-Chattext generationIQ3_XS8.8B7.29 GiB0.15 GiB
Mathstral-7B-v0.1text generationQ2_K7.2B7.37 GiB0.07 GiB
Yi-Coder-1.5B-Chattext generationIQ3_XS1.5B7.44 GiB0.00 GiB
Yi-1.5-6B-Chattext generationQ5_K_M6.1B6.83 GiB0.61 GiB
Ministral-3-3B-Instruct-2512-BF16text generationQ6_K_L4.3B6.78 GiB0.66 GiB
Mistral-7B-Instruct-v0.2text generationI1-Q2_K7.2B7.37 GiB0.07 GiB
Qwen2.5-VL-3B-Instructtext generationQ8_03.8B5.00 GiB2.44 GiB
○Spec sheet◐Predictedwhat these mean

This page models a generic 8GB accelerator, so it answers what fits rather than how fast it runs. For tokens per second you need a specific card — pick one from hardware, where bandwidth is known.