Guides · intermediate · 6 min · updated 2026-07-28

Three ways a model fails to fit, and what to do about each

Hard OOM, mid-context OOM, and the silent spill that runs 5-20x slow are different problems with different fixes. Diagnose first, then work down the ladder.

Diagnose before you fix

If the model loaded but generates at 2 tok/s, you have a spill, not an OOM, and no amount of tweaking sampler settings will help — some part of the model is in system RAM and every token waits on it. If it failed at load, you are usually a few hundred MiB short, and the cheapest fix is -ctk q8_0 -ctv q8_0, which cuts the KV cache to exactly 53.125% of f16 for very little quality cost. Try that before dropping a quant level.

First, though, check whether llama.cpp already changed your settings behind your back. --fit defaults to on. It reads free VRAM, projects the model's memory use, and adjusts anything you left unset to leave a 1024 MiB per-device margin. That is good behavior, but it means the config you ran may not be the config you typed.

The rules it follows are worth memorizing, because they decide which failure mode you get:

  • If you did not pass -c, it shrinks context first, down to a floor of 4096 (--fit-ctx changes the floor). Everything stays on the GPU and you get a short context.
  • If you did pass -c, it will not touch your context. It moves weights to the CPU instead. This is how you get a silent spill.
  • If you passed -ngl, -ot, --cpu-moe, or --n-cpu-moe, the fitter aborts entirely and warns failed to fit params to free device memory: n_gpu_layers already set by user to N, abort. You are on your own, which is often what you want.

So -c 65536 alone on a card that cannot hold it does not error. It quietly relocates layers and hands you a model that runs at a fraction of the speed. Pinning -ngl 99 turns that silence back into an honest crash.

Reading the memory breakdown

llama.cpp prints a memory table, but at trace verbosity — you will not see it by default. Add -lv 4 (or -v). Upstream's own example, on an RTX 4090:

llama_memory_breakdown_print: | memory breakdown [MiB] | total   free     self   model   context   compute    unaccounted |
llama_memory_breakdown_print: |   - CUDA0 (RTX 4090)   | 24077 =  945 + (19187 = 17904 +     384 +     898) +        3945 |
llama_memory_breakdown_print: |   - Host               |                 58271 = 58259 +       0 +      12                |

The equation is total = free + (self = model + context + compute) + unaccounted. Four things to read off it:

context is your KV cache. If this is the number that grew and killed you, shrink context or quantize the cache.

compute is scratch space for activations — 898 MiB here. It scales with the physical batch -ub (default 512, with logical -b at 2048), not with context length. Dropping -ub 512 to -ub 256 is a real, cheap way to reclaim a few hundred MiB when you are close.

unaccounted is 3945 MiB — memory on the card that is not llama.cpp's. Your compositor, your browser, another inference process, the CUDA context itself. This is the single most common answer to "why did it OOM when nvidia-smi showed enough free?"

The Host row is the tell for a spill. Host model of 58259 MiB means 57 GiB of weights are sitting in system RAM. If you expected a fully-offloaded model and the Host row is anything but near-zero, that is your 2 tok/s.

The fitter logs the same thing more directly when it relocates layers: - CUDA0 (NVIDIA GeForce RTX 4090): 48 layers (34 overflowing), 19187 MiB used, 1199 MiB free. "34 overflowing" is the string to grep for.

Why it OOMs with VRAM to spare

A hard OOM at load looks like ggml_backend_cuda_buffer_type_alloc_buffer: allocating 1234.00 MiB on device 0: cudaMalloc failed: out of memory. If free VRAM looked sufficient, the usual causes are the unaccounted column above, the compute buffer you forgot to budget, or fragmentation — allocations are rounded up 5% with 256-byte alignment, so a card with 400 MiB free in scattered holes cannot serve one 400 MiB request.

On Windows there is an extra trap. Since driver 536.40, the NVIDIA Control Panel setting CUDA - Sysmem Fallback Policy defaults to allowing allocations to spill into system RAM rather than fail. That converts a clean OOM into a silent slowdown. If you would rather crash than crawl, set it to Prefer No Sysmem Fallback. On Linux the equivalent is opt-in: GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 switches allocations to managed memory.

An OOM that happens partway through a long chat is almost always the KV cache reaching a length you never tested. Size for your worst case, not your first message.

The fix ladder

Work down this list. Each step costs more quality than the one above it.

1. Shorten context. Free, if you do not need the length. For Qwen3-8B (36 layers, 8 KV heads, head_dim 128) the cache is 144 KiB per token at f16 — 4.500 GiB at 32K, 1.125 GiB at 8K.

2. Quantize the KV cache. -ctk q8_0 -ctv q8_0. q8_0 stores 8.5 bits per element against f16's 16, so the cache drops to 53.125% — Qwen3-8B at 32K goes from 4.500 GiB to 2.391 GiB. This is the best value on the ladder. Two caveats: quantizing V requires flash attention, or context creation fails with V cache quantization requires flash_attn (-fa defaults to auto, so this usually just works); and q4_0 at 4.5 bits (28.125%) is a real quality drop, unlike q8_0. Allowed types are f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1.

3. Check you are not defeating a sliding-window cache. gpt-oss-120b alternates 18 sliding-window layers (window 128) with 18 full layers. llama.cpp sizes the SWA cache at pad256(min(ctx, n_swa + n_ubatch)) = 768 cells, so its KV at 128K is 4.526 GiB rather than the 9.000 GiB a flat 36-layer calculation predicts. Passing --swa-full throws that away and allocates the full size for every layer. Do not use it to "fix" anything.

4. Drop one quant level. Q5_K_M to Q4_K_M on Qwen3-8B saves about 1.1 GiB. Note that quantizing does not speed up prompt processing — prefill is compute-bound and Q4 is marginally slower there than F16, though decode is roughly 2.3x faster.

5. Offload MoE experts with --n-cpu-moe. For sparse models this is dramatically better than offloading whole layers, because expert weights are nearly all the bytes and only a fraction are read per token. In gpt-oss-120b's MXFP4 GGUF, the 687 tensors total 59.02 GiB and the exps tensors are 56.88 GiB of that — 96.37%. Everything else, attention and embeddings included, is 2.14 GiB. Each layer's experts are 1.58 GiB, so every --n-cpu-moe increment buys back that much VRAM. On a 24 GiB card at 32K context: 2.14 GiB non-expert + 1.15 GiB KV + ~0.9 GiB compute leaves room for roughly 12 layers of experts, so start at --n-cpu-moe 24 and walk down until it stops fitting. This is not PCIe-bound — weights are not streamed per token, only about 16 KB of activations cross each layer boundary.

6. Offload dense layers with -ngl. The blunt instrument, and the one that actually hurts. Reduce until the Host row goes to zero for the layers you kept.

7. Take the smaller model. A model that spills runs 5-20x slower, not slightly slower. A Q4 12B that fits entirely in VRAM beats a Q4 30B that half-fits, on both latency and usually on real output quality per unit of your patience.

Two things that will not save you

Adding a second GPU adds capacity, not speed: llama.cpp's layer split is sequential, so two 3090s give you 48 GB at roughly one card's token rate. vLLM's tensor parallelism does aggregate bandwidth, but pays an all-reduce, and consumer RTX cards have no NVLink to pay it over.

And --no-mmap is deprecated — it now warns and points you at --load-mode. It was never a memory fix for VRAM anyway; it changes how weights are read from disk, not where they end up.

Now see the numbers

More guides