Best local AI models for 24GB VRAM
Ranked by what actually fits at 32K context, computed from real file bytes.
A 24GB card gives you about 22.32 GiB to work with after driver overhead. 1867 indexed models fit at 32K context — the largest being Qwen3.5-99B at 99.0B parameters in I1-IQ1_S.
From the file· fit from summed bytesFrom the file· KV per layer
allvision + language 168text generation 1596speech recognition 39embeddings 26video generation 16speech synthesis 20image generation 2
Fits in 24GB at 32K context
largest quantization that fits, per model
| Model | Modality | Best quant | Params○ | Total◐ | Headroom◐ |
|---|---|---|---|---|---|
| Qwen3.6-35B-A3BMoE | vision + language | UD-Q4_K_S | 36.0B | 21.35 GiB | 0.97 GiB |
| embeddinggemma-300m | text generation | F32 | 303M | 2.05 GiB | 20.27 GiB |
| Qwen3.5-9B | vision + language | BF16 | 9.7B | 18.98 GiB | 3.34 GiB |
| gemma-4-26B-A4B-itMoE | vision + language | UD-Q5_K_M | 26.5B | 22.03 GiB | 0.29 GiB |
| nemotron-3.5-asr-streaming-0.6b | speech recognition | F32 | 638M | 3.22 GiB | 19.10 GiB |
| Qwen3-Coder-30B-A3B-InstructMoE | text generation | Q4_1 | 30.5B | 21.67 GiB | 0.65 GiB |
| Qwen3.5-4B | vision + language | BF16 | 4.7B | 9.88 GiB | 12.44 GiB |
| Qwythos-9B-Claude-Mythos-5-1M | vision + language | Q8_0 | 9.4B | 19.82 GiB | 2.50 GiB |
| gemma-4-12B-it-qat-q4_0-unquantized | text generation | Q4_0 | 12.0B | 9.81 GiB | 12.51 GiB |
| Qwen3-VL-30B-A3B-InstructMoE | vision + language | Q4_1 | 31.1B | 21.67 GiB | 0.65 GiB |
| gemma-4-26B-A4B-it-qat-q4_0-unquantizedMoE | vision + language | Q4_0 | 26.5B | 15.78 GiB | 6.54 GiB |
| Llama-3.2-1B-Instruct | text generation | F16 | 1.2B | 4.11 GiB | 18.21 GiB |
| Qwen-AgentWorld-35B-A3BMoE | text generation | UD-Q4_K_M | 34.7B | 22.04 GiB | 0.28 GiB |
| llama-3-youko-8b | text generation | Q8_0 | 8.0B | 12.79 GiB | 9.53 GiB |
| gpt-oss-20bMoE | text generation | F16 | 21.5B | 14.40 GiB | 7.92 GiB |
| Qwen3-8B | text generation | BF16 | 8.2B | 20.60 GiB | 1.72 GiB |
| Qwopus3.6-35B-A3B-v1MoE | vision + language | I1-Q4_1 | 36.0B | 21.78 GiB | 0.54 GiB |
| Qwen3.5-0.8B | vision + language | BF16 | 873M | 2.60 GiB | 19.72 GiB |
| gemma-4-e4b-it | vision + language | BF16 | 8.0B | 15.50 GiB | 6.82 GiB |
| Qwen3-4B | text generation | BF16 | 4.0B | 12.81 GiB | 9.51 GiB |
| UI-TARS-1.5-7B | text generation | F16 | 8.3B | 16.80 GiB | 5.52 GiB |
| Llama-3.1-8B-Instruct | text generation | BF16 | 8.0B | 19.81 GiB | 2.51 GiB |
| Ornith-1.0-35BMoE | text generation | UD-Q4_K_M | 34.7B | 22.04 GiB | 0.28 GiB |
| parakeet-tdt-0.6b-v3 | speech recognition | F32 | 627M | 3.18 GiB | 19.14 GiB |
| ThinkingCap-Qwen3.6-27B | vision + language | Q5_K_M | 27.4B | 22.19 GiB | 0.13 GiB |
| Llama-3.2-3B-Instruct | text generation | F16 | 3.2B | 10.30 GiB | 12.02 GiB |
| Qwopus3.6-27B-Coder | vision + language | Q5_K_M | 27.8B | 21.06 GiB | 1.26 GiB |
| whisper-medium | speech recognition | F32 | 764M | 3.69 GiB | 18.63 GiB |
| Qwythos-9B-v2 | vision + language | Q8_0 | 9.7B | 19.82 GiB | 2.50 GiB |
| gemma-4-E4B-it-qat-q4_0-unquantized | vision + language | Q4_0 | 7.9B | 6.12 GiB | 16.20 GiB |
| Voxtral-Mini-4B-Realtime-2602 | speech recognition | F16 | 4.4B | 12.33 GiB | 9.99 GiB |
| Wan2.1-T2V-1.3B | text generation | Q8_0 | 1.4B | 12.18 GiB | 10.14 GiB |
| Qwen3-Coder-NextMoE | text generation | IQ2_XXS | 79.7B | 21.76 GiB | 0.56 GiB |
| LFM2.5-1.2B-Instruct | text generation | BF16 | 1.2B | 3.37 GiB | 18.95 GiB |
| gemma-3-1b-it | text generation | F16 | 1000M | 2.81 GiB | 19.51 GiB |
| Qwen2.5-7B-Instruct | text generation | F16 | 7.6B | 16.80 GiB | 5.52 GiB |
| Qwen3-14B | text generation | Q8_0 | 14.8B | 20.48 GiB | 1.84 GiB |
| Qwen3-0.6B | text generation | BF16 | 752M | 5.68 GiB | 16.64 GiB |
| Qwen3.5-35B-A3BMoE | vision + language | Q4_K_M | 36.0B | 22.18 GiB | 0.14 GiB |
| Qwen2.5-Coder-7B-Instruct | text generation | Q8_0 | 7.6B | 17.69 GiB | 4.63 GiB |
| Qwen2.5-32B-Instruct | text generation | Q3_K_S | 32.8B | 22.30 GiB | 0.02 GiB |
| Qwen3-VL-4B-Instruct | vision + language | BF16 | 4.4B | 12.81 GiB | 9.51 GiB |
| Qwen2.5-1.5B-Instruct | text generation | F16 | 1.5B | 4.56 GiB | 17.76 GiB |
| Jan-v3-4B-base-instruct | text generation | BF16 | 4.4B | 13.53 GiB | 8.79 GiB |
| gemma-3-4b-it | text generation | BF16 | 4.3B | 8.85 GiB | 13.47 GiB |
| Qwen3.6-40B-Claude-4.6-Opus-Deckard-Heretic-Uncensored-Thinking | vision + language | IQ3_M | 39.5B | 20.98 GiB | 1.34 GiB |
| whisper-large-v3 | speech recognition | F16 | 1.5B | 3.74 GiB | 18.58 GiB |
| MiniCPM5-1B-Claude-Opus-Fable5-Thinking | text generation | F16 | 1.1B | 3.55 GiB | 18.77 GiB |
| Qwen3-30B-A3BMoE | text generation | Q4_1 | 30.5B | 21.69 GiB | 0.63 GiB |
| Qwen2.5-VL-7B-Instruct | vision + language | BF16 | 8.3B | 16.80 GiB | 5.52 GiB |
| Qwen3-VL-2B-Instruct | vision + language | BF16 | 2.1B | 7.50 GiB | 14.82 GiB |
| Ornith-1.0-9B | text generation | BF16 | 9.2B | 18.98 GiB | 3.34 GiB |
| Qwen3-1.7B | text generation | BF16 | 2.0B | 7.50 GiB | 14.82 GiB |
| Agents-A1MoE | text generation | Q4_K_M | 35.1B | 21.14 GiB | 1.18 GiB |
| gemma-4-26B-A4B-it-ultra-uncensored-hereticMoE | text generation | Q5_K_M | 25.8B | 20.15 GiB | 2.17 GiB |
| Qwen2.5-Coder-32B-Instruct | text generation | Q3_K_S | 32.8B | 22.30 GiB | 0.02 GiB |
| Qwen3.5-27B | vision + language | Q5_K_S | 27.8B | 21.39 GiB | 0.93 GiB |
| jina-embeddings-v5-text-small | embeddings | F16 | 596M | 5.39 GiB | 16.93 GiB |
| whisper-large-v3-turbo | speech recognition | F16 | 809M | 2.36 GiB | 19.96 GiB |
| Qwen2.5-Coder-14B-Instruct | text generation | Q6_K_L | 14.8B | 18.49 GiB | 3.83 GiB |
This page models a generic 24GB accelerator, so it answers what fits rather than how fast it runs. For tokens per second you need a specific card — pick one from hardware, where bandwidth is known.