Best local AI models for 10GB VRAM
Ranked by what actually fits at 32K context, computed from real file bytes.
A 10GB card gives you about 9.30 GiB to work with after driver overhead. 26 indexed models fit at 32K context — the largest being Nemotron-3-Embed-8B-BF16 at 8.0B parameters in IQ4_XS.
From the file· fit from summed bytesFrom the file· KV per layer
Fits in 10GB at 32K context
largest quantization that fits, per model
| Model | Modality | Best quant | Params○ | Total◐ | Headroom◐ |
|---|---|---|---|---|---|
| jina-embeddings-v5-text-small | embeddings | F16 | 596M | 5.39 GiB | 3.91 GiB |
| embeddinggemma-300m-qat-q8_0-unquantized | embeddings | Q8_0 | 303M | 1.22 GiB | 8.08 GiB |
| KaLM-embedding-multilingual-mini-instruct-v2.5 | embeddings | Q8_0 | 494M | 1.65 GiB | 7.65 GiB |
| nomic-embed-text-v1.5 | embeddings | F32 | 137M | 2.40 GiB | 6.90 GiB |
| jina-embeddings-v5-text-nano | embeddings | F16 | 212M | 2.30 GiB | 7.00 GiB |
| all-MiniLM-L6-v2 | embeddings | F32 | 23M | 1.13 GiB | 8.17 GiB |
| mxbai-embed-xsmall-v1 | embeddings | F32 | 24M | 1.13 GiB | 8.17 GiB |
| nomic-embed-text-v2-moeMoE | embeddings | F32 | 475M | 2.58 GiB | 6.72 GiB |
| bge-m3 | embeddings | Q8_0 | 567M | 4.37 GiB | 4.93 GiB |
| snowflake-arctic-embed-l-v2.0 | embeddings | F32 | 568M | 5.90 GiB | 3.40 GiB |
| jina-reranker-v1-tiny-en | embeddings | F16 | 33M | 1.01 GiB | 8.29 GiB |
| Qwen3.5-9B-DFlash | embeddings | BF16 | 1.3B | 3.47 GiB | 5.83 GiB |
| Qwen3-Embedding-0.6B | embeddings | F16 | 596M | 5.39 GiB | 3.91 GiB |
| gte-small | embeddings | Q8_0 | 33M | 1.36 GiB | 7.94 GiB |
| Qwen3-Embedding-8B | embeddings | Q3_K_L | 7.6B | 9.21 GiB | 0.09 GiB |
| nomic-embed-text-v1 | embeddings | F32 | 137M | 2.40 GiB | 6.90 GiB |
| LFM2.5-Embedding-350M | embeddings | F16 | 354M | 1.82 GiB | 7.48 GiB |
| Qwen3-Embedding-4B | embeddings | Q8_0 | 4.0B | 9.30 GiB | 0.00 GiB |
| Nemotron-3-Embed-8B-BF16 | embeddings | IQ4_XS | 8.0B | 9.20 GiB | 0.10 GiB |
| nomic-embed-code | embeddings | Q6_K_L | 7.1B | 8.14 GiB | 1.16 GiB |
| LCO-Embedding-Omni-3B-2605 | embeddings | Q8_0 | 4.7B | 5.30 GiB | 4.00 GiB |
| qwen-indic-v1 | embeddings | I1-IQ4_XS | 7.6B | 9.27 GiB | 0.03 GiB |
| Octen-Embedding-4B | embeddings | Q8_0 | 4.0B | 9.30 GiB | 0.00 GiB |
| bge-reranker-v2-m3 | embeddings | Q8_0 | 568M | 4.37 GiB | 4.93 GiB |
| LFM2.5-ColBERT-350M | embeddings | F16 | 353M | 1.82 GiB | 7.48 GiB |
| gte-large | embeddings | Q8_0 | 335M | 4.11 GiB | 5.19 GiB |
This page models a generic 10GB accelerator, so it answers what fits rather than how fast it runs. For tokens per second you need a specific card — pick one from hardware, where bandwidth is known.