What AI models can a Tesla V100 32GB run?

1949 of 2118 indexed open-weight models fit a Tesla V100 32GB at 65,536 context with q8_0 KV cache, the largest being Bernini-R at Q8_0. That covers text, vision-language, image, video and speech models.

How much usable memory does a Tesla V100 32GB actually have?

Its nameplate is 32 GB, but about 29.76 GiB is available to a model once driver and compositor overhead is accounted for.

Is a Tesla V100 32GB fast for local AI?

Its memory bandwidth is 900 GB/s, and that figure — not teraflops — is what governs token generation speed. Capacity decides what you can run; bandwidth decides how fast it runs.

NVIDIA · datacenter

Tesla V100 32GB

Tesla V100 32GB has 32 GB of VRAM at 900 GB/s — about 29.76 GiB usable after driver and compositor overhead. 1949 of 2118 indexed models fit at 64K context with q8_0 KV.

Spec sheet· bandwidth, theoreticalFrom the file· fit from summed bytesPredicted· speed

Memory

32 GB

HBM2

Bandwidth

900 GB/s

4096-bit bus

Tensor FP16

125 TF

dense

TDP

300 W

Context4K 8K 16K 32K 64K 128K

KV cachef16 q8_0 q4_0quantizing the KV cache is a ~2× lever on the dominant term at long context

video 16vision language 177text 1668embedding 26image 2audio tts 21audio asr 39

What fits at 64K context

largest quantization that fits, per model · 1949 of 2118 indexed

Model	Best quant	Params	Weights●	KV●	Total in memory◐	Headroom◐	tok/s◐
Bernini-R	Q8_0	14.3B	28.71 GiB	0.00 GiB	29.76 GiB	0.00 GiB	18±22%
Qwen3.6-40B-Claude-4.6-Opus-Deckard-Heretic-Uncensored-Thinking	Q5_K_S	39.5B	25.49 GiB	3.19 GiB	29.74 GiB	0.02 GiB	18±22%
MN-GRAND-23.5B-Gutenberg-UNCENSORED-V2-GLM4.7-Thinking	I1-Q6_K	23.4B	17.91 GiB	10.76 GiB	29.71 GiB	0.05 GiB	18±22%
Hunyuan-A13B-InstructMoE	IQ2_M	80.4B	24.45 GiB	4.25 GiB	29.69 GiB	0.07 GiB	18±22%
Apertus-70B-Instruct-2509	IQ2_XXS	70.6B	17.88 GiB	10.63 GiB	29.69 GiB	0.07 GiB	18±22%
WizardCoder-Python-34B-V1.0	I1-Q5_K_M	33.7B	22.20 GiB	6.38 GiB	29.67 GiB	0.09 GiB	18±22%
Phind-CodeLlama-34B-Python-v1	I1-Q5_K_M	33.7B	22.20 GiB	6.38 GiB	29.67 GiB	0.09 GiB	18±22%
Phind-CodeLlama-34B-v2	I1-Q5_K_M	33.7B	22.20 GiB	6.38 GiB	29.67 GiB	0.09 GiB	18±22%
CodeLlama-34b-instruct-hf	Q5_K_M	33.7B	22.20 GiB	6.38 GiB	29.67 GiB	0.09 GiB	18±22%
WizardLM-1.0-Uncensored-CodeLlama-34b	Q5_K_M	33.7B	22.20 GiB	6.38 GiB	29.67 GiB	0.09 GiB	18±22%
Qwen3.6-28BMoE	Q8_0	28.2B	28.00 GiB	0.66 GiB	29.67 GiB	0.09 GiB	83±37%
Phi-3.5-MoE-instructMoEKV unresolved	Q4_1	41.9B	24.41 GiB	4.25 GiB	29.66 GiB	0.10 GiB	34±37%
Darwin-35B-A3B-OpusMoE	Q6_K	36.0B	27.99 GiB	0.66 GiB	29.66 GiB	0.10 GiB	89±37%
Aurora-Code-1MoE	Q6_K	34.7B	27.99 GiB	0.66 GiB	29.66 GiB	0.10 GiB	89±37%
grug-35b-v2MoE	Q6_K	35.1B	27.99 GiB	0.66 GiB	29.66 GiB	0.10 GiB	89±37%
grug-35bMoE	Q6_K	35.1B	27.99 GiB	0.66 GiB	29.66 GiB	0.10 GiB	89±37%
WorldSim-Opus-3.6-35B-A3BMoE	Q6_K	35.1B	27.99 GiB	0.66 GiB	29.66 GiB	0.10 GiB	89±37%
Qwen3.6-35B-A3B-AnkoMoE	Q6_K	35.1B	27.99 GiB	0.66 GiB	29.66 GiB	0.10 GiB	89±37%
KAT-Coder-V2.5-DevMoE	Q6_K	34.7B	27.99 GiB	0.66 GiB	29.66 GiB	0.10 GiB	89±37%
Ornith-1.0-35BMoE	Q6_K	34.7B	27.99 GiB	0.66 GiB	29.66 GiB	0.10 GiB	89±37%
Nex-N2-miniMoE	Q6_K	35.1B	27.99 GiB	0.66 GiB	29.66 GiB	0.10 GiB	89±37%
Nemotron-Mini-4B-Instruct	F16	4.2B	24.38 GiB	4.25 GiB	29.65 GiB	0.11 GiB	18±22%
Skyfall-31B-v4.2	Q5_K_L	31.4B	21.35 GiB	7.17 GiB	29.64 GiB	0.12 GiB	18±22%
IQuest-Coder-V1-40B-Instruct	I1-Q3_K_M	39.8B	17.92 GiB	10.63 GiB	29.64 GiB	0.12 GiB	18±22%
Qwen3.5-99BMoE	I1-IQ2_S	99.0B	27.80 GiB	0.80 GiB	29.63 GiB	0.13 GiB	81±37%
Qwen3.6-35B-A3BMoE	UD-Q6_K	36.0B	27.95 GiB	0.66 GiB	29.62 GiB	0.14 GiB	89±37%
Homunculus	BF16	12.5B	23.21 GiB	5.31 GiB	29.57 GiB	0.19 GiB	18±22%
gemma-4-E4B-it-Uncensored-MAX	F32	8.0B	28.02 GiB	0.50 GiB	29.54 GiB	0.22 GiB	18±22%
Maenad-70B	I1-IQ2_XXS	70.6B	17.79 GiB	10.63 GiB	29.54 GiB	0.22 GiB	18±22%
DeepSeek-R1-Distill-Llama-70B-Uncensored-v2-Unbiased-Reasoner	I1-IQ2_XXS	70.6B	17.79 GiB	10.63 GiB	29.54 GiB	0.22 GiB	18±22%
Rombos-LLM-70b-Llama-3.3	I1-IQ2_XXS	70.6B	17.79 GiB	10.63 GiB	29.54 GiB	0.22 GiB	18±22%
L3.3-Electra-R1-70b	I1-IQ2_XXS	70.6B	17.79 GiB	10.63 GiB	29.54 GiB	0.22 GiB	18±22%
L3.3-70B-Magnum-v4-SE	IQ2_XXS	70.6B	17.79 GiB	10.63 GiB	29.54 GiB	0.22 GiB	18±22%
Latxa-Llama-3.1-70B-Instruct-v2	I1-IQ2_XXS	70.6B	17.79 GiB	10.63 GiB	29.54 GiB	0.22 GiB	18±22%
Llama-3.3_70_b_uncensored_continued	I1-IQ2_XXS	70.6B	17.79 GiB	10.63 GiB	29.54 GiB	0.22 GiB	18±22%
Llama-3.3-70B-Instruct-abliterated	I1-IQ2_XXS	70.6B	17.79 GiB	10.63 GiB	29.54 GiB	0.22 GiB	18±22%
grok-oss-Revenant-70B	I1-IQ2_XXS	70.6B	17.79 GiB	10.63 GiB	29.54 GiB	0.22 GiB	18±22%
Llama-3.1-Nemotron-70B-Instruct-HF	I1-IQ2_XXS	70.6B	17.79 GiB	10.63 GiB	29.54 GiB	0.22 GiB	18±22%
L3.3-70B-Euryale-v2.3	I1-IQ2_XXS	70.6B	17.79 GiB	10.63 GiB	29.54 GiB	0.22 GiB	18±22%
Hermes-3-Llama-3.1-70B	IQ2_XXS	70.6B	17.79 GiB	10.63 GiB	29.54 GiB	0.22 GiB	18±22%
Hermes-4-70B-heretic	I1-IQ2_XXS	70.6B	17.79 GiB	10.63 GiB	29.54 GiB	0.22 GiB	18±22%
Llama-3.3-70B-Instruct	IQ2_XXS	70.6B	17.79 GiB	10.63 GiB	29.54 GiB	0.22 GiB	18±22%
Llama-3.1-70B	IQ2_XXS	70.6B	17.79 GiB	10.63 GiB	29.54 GiB	0.22 GiB	18±22%
Anubis-70B-v1.2	IQ2_XXS	70.6B	17.79 GiB	10.63 GiB	29.54 GiB	0.22 GiB	18±22%
Hermes-4-70B	IQ2_XXS	70.6B	17.79 GiB	10.63 GiB	29.54 GiB	0.22 GiB	18±22%
Golem-70B-v1b	I1-IQ2_XXS	70.6B	17.79 GiB	10.63 GiB	29.54 GiB	0.22 GiB	18±22%
DeepSeek-R1-Distill-Llama-70B-abliterated	I1-IQ2_XXS	70.6B	17.79 GiB	10.63 GiB	29.54 GiB	0.22 GiB	18±22%
DeepSeek-R1-Distill-Llama-70B-heretic	I1-IQ2_XXS	70.6B	17.79 GiB	10.63 GiB	29.54 GiB	0.22 GiB	18±22%
DeepSeek-R1-Distill-Llama-70B	IQ2_XXS	70.6B	17.79 GiB	10.63 GiB	29.54 GiB	0.22 GiB	18±22%
Legion-V2.1-LLaMa-70B	I1-IQ2_XXS	70.6B	17.79 GiB	10.63 GiB	29.54 GiB	0.22 GiB	18±22%
Assistant_Pepe_70B	I1-IQ2_XXS	70.6B	17.79 GiB	10.63 GiB	29.54 GiB	0.22 GiB	18±22%
Tess-R1-Limerick-Llama-3.1-70B	IQ2_XXS	70.6B	17.79 GiB	10.63 GiB	29.54 GiB	0.22 GiB	18±22%
SEMIKONG-70B	IQ2_XXS	70.6B	17.79 GiB	10.63 GiB	29.54 GiB	0.22 GiB	18±22%
functionary-medium-v3.2KV unresolved	IQ2_XXS	70.6B	17.79 GiB	10.63 GiB	29.54 GiB	0.22 GiB	18±22%
Llama-3.1-WhiteRabbitNeo-2-70B	IQ2_XXS	70.6B	17.79 GiB	10.63 GiB	29.54 GiB	0.22 GiB	18±22%
Infinity-Instruct-7M-Gen-Llama3_1-70B	I1-IQ2_XXS	70.6B	17.79 GiB	10.63 GiB	29.54 GiB	0.22 GiB	18±22%
New-Dawn-Llama-3-70B-32K-v1.0	I1-IQ2_XXS	70.6B	17.79 GiB	10.63 GiB	29.54 GiB	0.22 GiB	18±22%
Meta-Llama-3-70B-Instruct-abliterated-v3.5	I1-IQ2_XXS	70.6B	17.79 GiB	10.63 GiB	29.54 GiB	0.22 GiB	18±22%
Athene-70B	IQ2_XXS	70.6B	17.79 GiB	10.63 GiB	29.54 GiB	0.22 GiB	18±22%
L3.3-70B-Magnum-Diamond	IQ2_XXS	70.6B	17.79 GiB	10.63 GiB	29.54 GiB	0.22 GiB	18±22%

●From the file◐Predictedwhat these mean

Speed is modeled, not measured: decode is memory-bandwidth bound, so tokens per second is bytes read per token against achievable bandwidth. Mixture-of-experts models carry a wider band because only the routed experts are read each step, and few have been measured publicly.

Questions people ask

What AI models can a Tesla V100 32GB run?: 1949 of 2118 indexed open-weight models fit a Tesla V100 32GB at 65,536 context with q8_0 KV cache, the largest being Bernini-R at Q8_0. That covers text, vision-language, image, video and speech models.
How much usable memory does a Tesla V100 32GB actually have?: Its nameplate is 32 GB, but about 29.76 GiB is available to a model once driver and compositor overhead is accounted for.
Is a Tesla V100 32GB fast for local AI?: Its memory bandwidth is 900 GB/s, and that figure — not teraflops — is what governs token generation speed. Capacity decides what you can run; bandwidth decides how fast it runs.