Guides
Written for people who actually run this stuff. Every number in these guides is one we verified, and where something genuinely isn't known we say so rather than filling the gap with a plausible figure.
Running your first local model
Pick a runtime, pick a GGUF file that actually fits your VRAM, and know how to tell whether it fit.
Choosing a GGUF quantization
Q4_K_M is a good default and Q5_K_M or Q6_K if it fits — but the per-model data needed to do better than that is not published anywhere.
Why long context eats your VRAM, and when it doesn't
KV cache is sized by num_key_value_heads and allocated up front. For Gemma-3, DeepSeek, and Qwen3.5/3.6 the flat formula overstates it 4x to 71x.
Running a model bigger than your VRAM
MoE models offload to system RAM well because all experts must be resident but only a few are read per token. Here is the arithmetic and the llama.cpp flags.
Choosing a GPU for local inference
Capacity decides what you can run; memory bandwidth decides how fast. Teraflops barely matter for generation, and your current card may already be enough.
Apple Silicon for Local AI: What a Mac Actually Buys You
Unified memory buys capacity no consumer GPU can match, but prefill is 3-4x slower and the same chip name ships at two bandwidths 33% apart.
What you actually need to run FLUX, Wan, and SDXL locally
A diffusion model is a pipeline of four models, not one file, and the headline parameter count covers only the denoiser. Here is the real memory math.
Running Speech Models Locally: ASR and TTS on Hardware You Already Own
Voice models are 60 MB to 3 GB, not 30 GB. Almost any laptop runs them, but the RTF figures you've read were measured on an H200 at batch 64.
A second GPU buys capacity, not speed
llama.cpp's default layer split runs your GPUs one at a time, so two 3090s give you 48GB at roughly one card's token rate. Here is the arithmetic.
A vision model is two files, and images cost more context than you expect
VLMs need a separate mmproj file alongside the main GGUF, and one image can burn 4096 context tokens — 576 MiB of KV cache on Qwen3-VL-8B.
Choosing a local model runtime
Most of these are layers over llama.cpp, not rivals to it. Pick a wrapper for ergonomics; switch to vLLM only when you serve concurrent requests.
How much worse is Q4? What quantization costs, and what nobody measured
On the one model with public measurements, Q4_K_M costs 2.4% perplexity. The step from Q4 to Q3 costs more than everything above it combined.
Local Embeddings and Rerankers: The Model Is Never the Bottleneck
Retrieval models run from 91 MiB to 1.1 GiB and work fine on CPU. Your real costs are vector storage, the reranker pass, and MTEB's doubled memory field.
Three ways a model fails to fit, and what to do about each
Hard OOM, mid-context OOM, and the silent spill that runs 5-20x slow are different problems with different fixes. Diagnose first, then work down the ladder.