Choosing a local model runtime

Start here

One person talking to one model: llama.cpp, or anything wrapping it. Ollama for a daemon and a registry, LM Studio for a GUI. On Apple silicon try MLX alongside whichever you pick, because it is a genuinely different implementation and not a skin. Move to vLLM when you have concurrent users on a CUDA or ROCm GPU with VRAM enough to hold the weights in a format it likes. That is the entire decision.

What makes this confusing is treating these as five competitors. Three are the same engine.

Ollama and LM Studio are wrappers, and you can read the pin

Ollama's repository root contains a file called LLAMA_CPP_VERSION. Today it reads b10091. Its build system is explicit about what that means — from CMakeLists.txt:

GGML backend for inference is provided by llama-server (built separately via llama/server/CMakeLists.txt using FetchContent from the pinned llama.cpp source).

So Ollama fetches upstream llama.cpp at a pinned tag, applies its patches from llama/compat/, builds llama-server, and drives it. Worth stating plainly, because the widespread claim that "Ollama wrote its own engine" is half true at best. It did build one for multimodal models, and its own blog says that engine still sits on "the tensor library that powers Ollama's inference" — ggml, the same library. For text models it runs upstream's server binary.

The pin is also how you measure the lag instead of guessing. Tag b10091 is commit b4d6c7d8ff69, dated 2026-07-22. The latest upstream llama.cpp release as of this writing is b10156, published 2026-07-28. Sixty-five builds, six days. That number will be wrong by the time you read this, which is the point — check it yourself:

curl -s https://raw.githubusercontent.com/ollama/ollama/main/LLAMA_CPP_VERSION
curl -s https://api.github.com/repos/ggml-org/llama.cpp/releases/latest | grep tag_name

Six days is not the crippling lag Ollama's critics describe. It is also not zero, and if you are chasing a model released this week, the diff between those tags is what you are missing.

LM Studio is the same arrangement with a GUI and honest plumbing: lms runtime ls, get, select, and update manage swappable engines, so you can move a runtime version without moving the app. Its Apple path is a separate open-source project, lmstudio-ai/mlx-engine, built on mlx-lm plus mlx-vlm, requiring macOS 14.0.

What transfers is GGUF, with one asterisk

llama.cpp, Ollama, and LM Studio all load the same GGUF files, and Ollama will adopt an external one rather than re-downloading its own copy — write a Modelfile containing FROM ./your-file.gguf and run ollama create my-model.

MLX is where the chain breaks. mlx-lm loads safetensors, not GGUF. It ships a gguf.py, but that module is export-only — convert_to_gguf() calls mx.save_gguf(), and its GGMLFileType enum has exactly one member, GGML_TYPE_F16. You can write a 16-bit GGUF out of MLX. You cannot read a quantized GGUF into it. An MLX Mac keeps a second copy of every model.

Those copies are not the same size, and the differences are larger than the labels suggest. Qwen3-8B is 8,190,735,360 parameters; every byte count below came from the HuggingFace API or the Ollama registry this session.

Three files that all answer to "4-bit Qwen3-8B" span 617,539,322 bytes — 589 MiB, enough to decide whether an 8 GB card holds the thing. MLX lands nearest its nominal width because its quantization is uniform: that repo's config.json declares group_size: 64, bits: 4, and 4 bits per weight plus a 16-bit scale and 16-bit bias per group of 64 is 4 + 32/64 = 4.5 bpw exactly, which is what the file measures. GGUF's K-quants promote sensitive tensors and overshoot their label. Neither is wrong, but only one matches the arithmetic people do in their heads.

At 8-bit the gap closes to 0.007 bpw. The format argument is a 4-bit argument.

Ollama's tags are a menu, not a dial

qwen3:8b, qwen3:8b-q4_K_M, and qwen3:latest all resolve to digest a3de86cd1c13 — one 5,225,374,496-byte blob. The default is Q4_K_M, which is reasonable. But qwen3:8b-q8_0 (8,851,075,872 bytes) and qwen3:8b-fp16 (16,388,043,552) exist while qwen3:8b-q6_K simply does not. You get the quants the registry publishes; llama.cpp's -hf and LM Studio's file browser give you every file in the repo. If Q6_K or IQ4_XS is what fits your card, that decides your runtime.

Concurrency, and the flag that surprises people

The common claim that llama.cpp can't batch is false. llama-server has had continuous batching on by default for a long time — -cb is enabled unless you pass -nocb — and -np sets the slot count.

What differs is KV cache allocation, and llama.cpp's default here is subtle. From tools/server/server.cpp:

if (params.n_parallel < 0) {
    SRV_TRC("%s", "n_parallel is set to auto, using n_parallel = 4 and kv_unified = true\n");
    params.n_parallel = 4;
    params.kv_unified = true;
}

Leave -np alone and you get four slots sharing one unified cache. Set it explicitly and kv_unified stays false, at which point llama-context.cpp does n_ctx_seq = n_ctx / n_seq_max. So -c 32768 -np 4 gives each request 8192 tokens, not 32768, and long prompts start failing for reasons the flags do not advertise. Pass -kvu to keep the shared pool, or multiply -c by your slot count.

vLLM's answer is paged blocks allocated on demand, so an idle slot costs nothing and shared prefixes are deduplicated — enable_prefix_caching defaults to True. That is the real architectural gap, and why vLLM wins under load and loses for one user. It reserves 92% of your GPU at startup (gpu_memory_utilization default 0.92) whether you need it or not, and captures CUDA graphs before serving unless you pass --enforce-eager. A single chat session pays all that overhead for throughput it never uses.

vLLM also wants different weights. Its GGUF support is documented as "highly experimental and under-optimized at the moment, it might be incompatible with other features." The supported paths are AWQ, GPTQ, bitsandbytes, and compressed-tensors formats like FP8 W8A8 and INT4 W4A16. And on Apple silicon, core vLLM lists you under CPU; GPU acceleration lives in a separate out-of-tree vLLM-Metal project. A Mac is not a vLLM machine.

A claim we could not confirm

"New architectures land in llama.cpp first" is repeated everywhere, and we expected to repeat it here. Checking it today, it does not hold. llama.cpp's LLM_ARCH_NAMES map lists 138 architectures; vLLM's _TEXT_GENERATION_MODELS registry lists 127. Those are counted differently — architecture families versus HuggingFace class names — so the gap itself means nothing. What matters is that on the newest entries they are at parity: Gemma 4, DeepSeek-V4, MiniMax-M3, Kimi-Linear, and Qwen3.5 are in both. The lag is downstream, in the wrappers and in how long someone takes to publish a good quant, not between the two engines.

The runtime can move throughput more than the model

The clearest published example is in speech, because its authors state their conditions. F5-TTS benchmarks one L20, 26 prompt/text pairs, 16 NFE steps, and reports RTF 0.0402 under TensorRT-LLM against 0.1467 under PyTorch. Same weights, same GPU, same audio, 3.6x from the runtime alone.

Nothing equivalent exists for llama.cpp against MLX on identical hardware, and we have not measured it, so this page contains no number for it. Treat the F5-TTS result as evidence for the shape of the effect, not its size: before concluding a model is too slow for your machine, run it under a second runtime. That test costs an evening and occasionally makes the hardware question disappear.

Practical defaults

llama.cpp when you want to know exactly what ran and reach new flags first. Ollama for an OpenAI-compatible endpoint on :11434 you never think about again, accepting the pin and the tag menu. LM Studio on :1234 when comparing quants visually beats scripting. mlx_lm.server on :8080 for Apple silicon experiments, keeping its own warning in mind: "not recommended for production as it only implements basic security checks." vLLM when requests arrive in parallel and the GPU says NVIDIA or AMD.

If you are unsure, you are not a vLLM user yet. Concurrency is the switch, and you will know when you have flipped it.