vLLM
Built for serving many concurrent requests. Much better throughput under load, and the wrong tool for one person chatting.
From the file· capabilities, not benchmarks
What it is
A serving engine using paged attention and continuous batching to keep a GPU busy across many simultaneous requests. Throughput scales with concurrency in a way single-user engines do not.
Capabilities
| Model formats | safetensors, AWQ, GPTQ, FP8, compressed-tensors |
| KV cache quantization | yes |
| CPU offload | not supported |
| MoE expert offload | not supported |
| Multi-GPU | Tensor parallelism — genuinely aggregates bandwidth, unlike a layer split. |
| Concurrency | This is the entire point. Hundreds of concurrent requests on one GPU. |
| Platforms | Linux |
Best for
Serving an application or a team, where many requests arrive at once.
Watch out for
It does not read GGUF in any practical sense, wants the whole model resident, and has no useful CPU offload. For a single user on a consumer card it will usually be slower and pickier than llama.cpp.