What AI models can a RTX 5000 Ada Generation run?

2024 of 2118 indexed open-weight models fit a RTX 5000 Ada Generation at 4,096 context with q8_0 KV cache, the largest being Bernini-R at Q8_0. That covers text, vision-language, image, video and speech models.

How much usable memory does a RTX 5000 Ada Generation actually have?

Its nameplate is 32 GB, but about 29.76 GiB is available to a model once driver and compositor overhead is accounted for.

Is a RTX 5000 Ada Generation fast for local AI?

Its memory bandwidth is 576 GB/s, and that figure — not teraflops — is what governs token generation speed. Capacity decides what you can run; bandwidth decides how fast it runs.

NVIDIA · workstation

RTX 5000 Ada Generation

RTX 5000 Ada Generation has 32 GB of VRAM at 576 GB/s — about 29.76 GiB usable after driver and compositor overhead. 2024 of 2118 indexed models fit at 4K context with q8_0 KV.

Spec sheet· bandwidth, theoreticalFrom the file· fit from summed bytesPredicted· speed

Memory

32 GB

GDDR6

Bandwidth

576 GB/s

256-bit bus

Tensor FP16

261 TF

dense

TDP

250 W

$4000 MSRP

Context4K 8K 16K 32K 64K 128K

KV cachef16 q8_0 q4_0quantizing the KV cache is a ~2× lever on the dominant term at long context

video 16text 1739vision language 181image 2embedding 26audio tts 21audio asr 39

What fits at 4K context

largest quantization that fits, per model · 2024 of 2118 indexed

Model	Best quant	Params	Weights●	KV●	Total in memory◐	Headroom◐	tok/s◐
Bernini-R	Q8_0	14.3B	28.71 GiB	0.00 GiB	29.76 GiB	0.00 GiB	12±22%
Huihui-Qwen3-Coder-Next-abliteratedMoE	I1-IQ3_XXS	79.7B	28.68 GiB	0.05 GiB	29.72 GiB	0.04 GiB	79±37%
CalmeRys-78B-Orpo-v0.1	I1-IQ2_S	78.0B	27.87 GiB	0.71 GiB	29.71 GiB	0.05 GiB	12±22%
Salience-1.5-ProMoE	Q6_K_L	36.0B	28.66 GiB	0.04 GiB	29.71 GiB	0.05 GiB	70±37%
Qwable-v1MoE	Q6_K_L	36.0B	28.66 GiB	0.04 GiB	29.71 GiB	0.05 GiB	70±37%
T-SearchMoE	Q6_K_L	36.0B	28.66 GiB	0.04 GiB	29.71 GiB	0.05 GiB	70±37%
v6-Finch-14B-HF	F16	14.1B	26.63 GiB	2.03 GiB	29.70 GiB	0.06 GiB	12±22%
Gemma-3-27B-MeditronFO	Q8_0	28.8B	28.13 GiB	0.49 GiB	29.70 GiB	0.06 GiB	12±22%
CodeLlama-70b-Instruct-hf	I1-IQ3_S	69.0B	27.86 GiB	0.66 GiB	29.65 GiB	0.11 GiB	12±22%
CodeLlama-70b-Python-hf	I1-IQ3_S	69.0B	27.86 GiB	0.66 GiB	29.65 GiB	0.11 GiB	12±22%
Nous-Hermes-Llama2-70b	I1-IQ3_S	69.0B	27.86 GiB	0.66 GiB	29.65 GiB	0.11 GiB	12±22%
Midnight-Miqu-70B-v1.5	I1-IQ3_S	69.0B	27.86 GiB	0.66 GiB	29.65 GiB	0.11 GiB	12±22%
KafkaLM-70B-German-V0.1	Q3_K_S	69.0B	27.86 GiB	0.66 GiB	29.65 GiB	0.11 GiB	12±22%
llama2_70b_chat_uncensored	Q3_K_S	69.0B	27.86 GiB	0.66 GiB	29.65 GiB	0.11 GiB	12±22%
Xwin-LM-70b-V0.1	Q3_K_S	69.0B	27.86 GiB	0.66 GiB	29.65 GiB	0.11 GiB	12±22%
Llama-2-70b-chat-hf	Q3_K_S	69.0B	27.86 GiB	0.66 GiB	29.65 GiB	0.11 GiB	12±22%
Seed-OSS-36B-Instruct	Q6_K_L	36.2B	27.99 GiB	0.53 GiB	29.62 GiB	0.14 GiB	12±22%
Hermes-4.3-36B	Q6_K_L	36.2B	27.99 GiB	0.53 GiB	29.62 GiB	0.14 GiB	12±22%
Melody1437-27B	Q3_K_M	27.8B	28.40 GiB	0.13 GiB	29.60 GiB	0.16 GiB	12±22%
Rombo-LLM-V3.0-Qwen-72b	I1-Q2_K	72.7B	27.76 GiB	0.66 GiB	29.56 GiB	0.20 GiB	12±22%
Qwen2.5-72B-Instruct-abliterated	I1-Q2_K	72.7B	27.76 GiB	0.66 GiB	29.56 GiB	0.20 GiB	12±22%
Qwen2.5-72B-Instruct-abliterated-v2	I1-Q2_K	72.7B	27.76 GiB	0.66 GiB	29.56 GiB	0.20 GiB	12±22%
HuatuoGPT-o1-72B	Q2_K	72.7B	27.76 GiB	0.66 GiB	29.56 GiB	0.20 GiB	12±22%
MiroThinker-v1.0-72B	I1-Q2_K	72.7B	27.76 GiB	0.66 GiB	29.56 GiB	0.20 GiB	12±22%
EVA-Qwen2.5-72B-v0.2	Q2_K	72.7B	27.76 GiB	0.66 GiB	29.56 GiB	0.20 GiB	12±22%
Qwen2.5-Math-72B-Instruct	Q2_K	72.7B	27.76 GiB	0.66 GiB	29.56 GiB	0.20 GiB	12±22%
Qwen2.5-72B-Instruct	Q2_K	72.7B	27.76 GiB	0.66 GiB	29.56 GiB	0.20 GiB	12±22%
Malaysian-Qwen2.5-72B-Instruct	I1-Q2_K	72.7B	27.76 GiB	0.66 GiB	29.56 GiB	0.20 GiB	12±22%
Qwen2.5-72B	I1-Q2_K	72.7B	27.76 GiB	0.66 GiB	29.56 GiB	0.20 GiB	12±22%
magnum-v4-72b	I1-Q2_K	72.7B	27.76 GiB	0.66 GiB	29.56 GiB	0.20 GiB	12±22%
KAT-Dev-72B-Exp	Q2_K	72.7B	27.76 GiB	0.66 GiB	29.56 GiB	0.20 GiB	12±22%
Homer-v1.0-Qwen2.5-72B	Q2_K	72.7B	27.76 GiB	0.66 GiB	29.56 GiB	0.20 GiB	12±22%
Qwen2.5-VL-72B-Instruct	Q2_K	73.4B	27.76 GiB	0.66 GiB	29.56 GiB	0.20 GiB	12±22%
Chuluun-Qwen2.5-72B-v0.01	Q2_K	72.7B	27.76 GiB	0.66 GiB	29.56 GiB	0.20 GiB	12±22%
Tower-Plus-72B-ultra-uncensored-heretic	I1-Q2_K	72.7B	27.76 GiB	0.66 GiB	29.56 GiB	0.20 GiB	12±22%
Chronos-Platinum-72B	Q2_K	72.7B	27.76 GiB	0.66 GiB	29.56 GiB	0.20 GiB	12±22%
UI-TARS-72B-DPO	Q2_K	73.4B	27.76 GiB	0.66 GiB	29.56 GiB	0.20 GiB	12±22%
Qwen2.5-7B-Instruct-1M	F32	7.6B	28.38 GiB	0.12 GiB	29.55 GiB	0.21 GiB	12±22%
DeepSeek-R1-Distill-Qwen-7B	F32	7.6B	28.38 GiB	0.12 GiB	29.55 GiB	0.21 GiB	12±22%
UI-TARS-7B-DPO	F32	8.3B	28.38 GiB	0.12 GiB	29.55 GiB	0.21 GiB	12±22%
Qwen2-7B-Instruct	F32	7.6B	28.38 GiB	0.12 GiB	29.55 GiB	0.21 GiB	12±22%
Hercules-5.0-Qwen2-7B	F32	7.6B	28.38 GiB	0.12 GiB	29.55 GiB	0.21 GiB	12±22%
Kepler-8B-Instruct-v2	F16	7.6B	28.37 GiB	0.12 GiB	29.54 GiB	0.22 GiB	12±22%
MiniCPM-o-2_6	F32	8.7B	28.37 GiB	0.12 GiB	29.54 GiB	0.22 GiB	12±22%
Llama-4-Scout-17B-16E-InstructMoEKV unresolved	IQ2_XXS	109B	28.09 GiB	0.40 GiB	29.52 GiB	0.24 GiB	49±37%
Qwen3-72B-Synthesis	Q2_K	72.7B	27.68 GiB	0.66 GiB	29.48 GiB	0.28 GiB	12±22%
Qwen3-53B-A3B-2507-THINKING-TOTAL-RECALL-v2-MASTER-CODERMoE	I1-Q4_K_S	53.0B	28.13 GiB	0.35 GiB	29.48 GiB	0.28 GiB	45±37%
Devstral-2-123B-Instruct-2512	IQ1_M	125B	27.59 GiB	0.73 GiB	29.47 GiB	0.29 GiB	12±22%
Mistral-Medium-3.5-128B	I1-IQ1_M	128B	27.59 GiB	0.73 GiB	29.47 GiB	0.29 GiB	12±22%
XORTRON-NXTXPRTXXL	I1-IQ1_M	128B	27.59 GiB	0.73 GiB	29.47 GiB	0.29 GiB	12±22%
Apertus-70B-Instruct-2509	IQ3_XS	70.6B	27.55 GiB	0.66 GiB	29.40 GiB	0.36 GiB	12±22%
Qwen3.5-88BMoE	I1-Q2_K_S	87.7B	28.30 GiB	0.05 GiB	29.37 GiB	0.39 GiB	62±37%
Snowpiercer-15B-v4	BF16	15.0B	27.90 GiB	0.42 GiB	29.36 GiB	0.40 GiB	12±22%
Kimi-Dev-72B	UD-IQ2_M	72.7B	27.56 GiB	0.66 GiB	29.36 GiB	0.40 GiB	12±22%
Kimi-Linear-48B-A3B-InstructMoE	Q4_K_L	49.1B	28.26 GiB	0.06 GiB	29.33 GiB	0.43 GiB	12±22%
Qwen3-42B-A3B-2507-Thinking-Abliterated-uncensored-TOTAL-RECALL-v2-Medium-MASTER-CODERMoE	I1-Q5_K_M	42.4B	28.05 GiB	0.28 GiB	29.33 GiB	0.43 GiB	51±37%
deepseek-llm-67b-chat	I1-IQ3_S	67.4B	27.40 GiB	0.79 GiB	29.28 GiB	0.48 GiB	12±22%
deepseek-llm-67b-base	I1-IQ3_S	67.4B	27.40 GiB	0.79 GiB	29.28 GiB	0.48 GiB	12±22%
openbuddy-deepseek-67b-v15.3-4k	I1-IQ3_S	67.4B	27.39 GiB	0.79 GiB	29.28 GiB	0.48 GiB	12±22%
Darwin-35B-A3B-OpusMoE	Q6_K_L	36.0B	28.22 GiB	0.04 GiB	29.26 GiB	0.50 GiB	71±37%

●From the file◐Predictedwhat these mean

Speed is modeled, not measured: decode is memory-bandwidth bound, so tokens per second is bytes read per token against achievable bandwidth. Mixture-of-experts models carry a wider band because only the routed experts are read each step, and few have been measured publicly.

Questions people ask

What AI models can a RTX 5000 Ada Generation run?: 2024 of 2118 indexed open-weight models fit a RTX 5000 Ada Generation at 4,096 context with q8_0 KV cache, the largest being Bernini-R at Q8_0. That covers text, vision-language, image, video and speech models.
How much usable memory does a RTX 5000 Ada Generation actually have?: Its nameplate is 32 GB, but about 29.76 GiB is available to a model once driver and compositor overhead is accounted for.
Is a RTX 5000 Ada Generation fast for local AI?: Its memory bandwidth is 576 GB/s, and that figure — not teraflops — is what governs token generation speed. Capacity decides what you can run; bandwidth decides how fast it runs.