Running your first local model

Start here

Install Ollama and run ollama run qwen3:8b. That downloads 5,225,374,496 bytes and gives you a chat prompt. If it's fast, you're done and you can read the rest of this later.

The rest of this page exists because of the two things that command hides: which file you actually got, and whether it fit in VRAM. Those are the only two questions that matter at this stage. Runtime choice is mostly preference; all three options below load the same GGUF format and share most of the same inference code underneath.

The three runtimes

llama.cpp is where features land first, so a model released last week is supported there before anywhere else. Ollama is the best default if you want an OpenAI-compatible endpoint running as a service and never want to think about it again. LM Studio is the best way to browse quants, because it shows you every file in a repo with its size before you commit to a download.

Sizing a model to your hardware

The rule: the model file should be roughly your VRAM minus 2 to 3 GiB. That headroom covers the KV cache, compute buffers, and whatever your desktop compositor is using. It's a starting heuristic, not a measurement, because KV cache size varies by more than an order of magnitude between architectures. Gemma-3-27B at 32K context needs 3.109 GiB of KV, while a flat parameter-count formula would tell you 15.50 GiB. Don't trust a formula for that part; check the loader's own output, which is covered below.

Some real file sizes, taken from the HuggingFace API rather than estimated:

So: 8 GB card, Qwen3-8B at Q4_K_M. 12 GB, gemma-3-12b at Q4_K_M or Qwen3-8B at Q8_0. 16 GB, either of those with room to spare. 24 GB, a 30B mixture-of-experts model like Qwen3-30B-A3B, which activates only about 3B parameters per token and is therefore much faster than its size suggests.

One number worth internalising: that "4-bit" Qwen3-8B file is 5,027,784,224 bytes for 8.19B parameters, which is 4.911 bits per weight, not 4.5. Q4_K_M is a mixture. Parsing its header gives 399 tensors: 217 at Q4_K, 145 at F32, and 37 at Q6_K. Attention output and feed-forward down-projection tensors get promoted to higher precision because quantizing them hurts more. Every "size = params × bits ÷ 8" calculator you'll find is wrong for this reason, usually by 5-10%.

Reading a GGUF filename

Take bartowski/Qwen_Qwen3-8B-GGUF and the file Qwen_Qwen3-8B-Q4_K_M.gguf.

bartowski is the repacker, not the model author. He and unsloth are the two most common, and both publish imatrix-calibrated quants. Qwen_Qwen3-8B is the base model with the original org name flattened into the filename. Q4_K_M is the quant: Q4 is the nominal bit width, K means K-quants (block-wise, with per-block scales), and the S/M/L suffix sets how aggressively sensitive tensors get promoted. IQ is a different, newer family that gets more quality out of very low bit widths at some speed cost. Unsloth's UD- prefix marks their own dynamic recipe.

Large models are split: unsloth/Qwen3-235B-A22B-GGUF stores UD-Q4_K_XL/Qwen3-235B-A22B-UD-Q4_K_XL-00001-of-00003.gguf plus two siblings, 134,123,851,136 bytes total, or 124.91 GiB. Download all shards and point the loader at 00001 only. It finds the rest. Grabbing just the first shard and wondering why it fails is the most common mistake here.

Actually running it

llama.cpp will fetch from HuggingFace directly, so you don't need to download anything by hand:

llama-server -hf bartowski/Qwen_Qwen3-8B-GGUF:Q4_K_M -c 8192

Then open http://127.0.0.1:8080. If you omit :Q4_K_M, -hf defaults to Q4_K_M anyway and falls back to the first file in the repo if that quant doesn't exist. Recent builds default -ngl to auto and --fit on, which shrinks unset arguments until the model fits in device memory, leaving a 1024 MiB margin per device. This is genuinely helpful, but note the word "unset": if you explicitly pass -c 32768, it will not reduce it for you, and you can still OOM.

The Ollama equivalent, with timings printed after each response:

ollama run --verbose qwen3:8b

LM Studio, from the terminal:

lms get qwen3-8b@q4_k_m
lms load qwen3-8b
lms server start

It ran, but it's slow

First check whether it actually fit. This is not a guess, both runtimes will tell you.

llama.cpp prints, during load, a model buffer size line per device and an offloaded N/M layers to GPU line. If N is less than M, part of the model is in system RAM. It also prints a KV buffer size line, which is your real KV cache cost for the context you asked for.

Ollama has ollama ps, whose PROCESSOR column reads 100% GPU, 100% CPU, or a split like 23%/77% CPU/GPU. Anything other than 100% GPU means you're spilling.

Spilling is not a small penalty, and this is where most people's mental model is wrong. Token generation is bandwidth-bound: the cost per token is essentially "read every active weight once." An RTX 3090 reads at 936 GB/s. Dual-channel DDR5-5600 tops out at 89.6 GB/s theoretical, about 10.4x slower, and DDR4-3200 at 51.2 GB/s is 18.3x slower. Splitting the weights splits the time:

Spilling one tenth of the model roughly halves your speed. These figures are lower bounds, derived from theoretical peak DRAM bandwidth, and real CPU inference doesn't reach peak. So the fix is almost never "tune something." It's "use a smaller quant, or a shorter context, so it fits."

If it fits and it's still slow, the likely cause is that you're measuring prompt processing rather than generation. Those are different regimes: prefill is compute-bound, and quantizing does not speed it up. Q4 is consistently a bit slower than F16 on prefill because of dequantization cost, even while decode at Q4 runs about 2.3x faster.

It OOMed

Reduce in this order, because this is roughly the order of "quality lost per GiB saved":

1. Cut context. -c 4096 instead of -c 32768. Costs you nothing if your prompts are short.
2. Quantize the KV cache. -ctk q8_0 -ctv q8_0 in llama.cpp. q8_0 is 8.5 bits per element versus f16's 16, so this roughly halves KV cost. Quality impact at q8_0 is small; at q4_0 (4.5 bits) it is noticeable.
3. Drop one quant level. Q4_K_M to IQ4_XS on Qwen3-8B saves 465,944,576 bytes, about 0.43 GiB.
4. Only then offload layers to CPU, and accept the table above.

For MoE models specifically there's a better lever: --n-cpu-moe N keeps the expert weights of the first N layers in system RAM while all the dense parts stay on the GPU. Experts are the bulk of the bytes but only a fraction get read per token, so this trades far less speed than generic layer offloading.

The thing about Ollama's names

ollama run qwen3:8b does not tell you what you're getting. The manifest does: qwen3:8b, qwen3:8b-q4_K_M, and qwen3:latest are all the same digest, a 5,225,374,496-byte blob. So the default is Q4_K_M. Fine.

Except that bartowski's Qwen3-8B Q4_K_M is 5,027,784,224 bytes. Both files declare general.file_type = 15, which is LLAMA_FTYPE_MOSTLY_Q4_K_M in llama.cpp's header. Both contain exactly 399 tensors. They differ by 197,590,272 bytes, and diffing the headers shows why: all 36 blk.N.attn_v.weight tensors are stored at F16 in Ollama's build, where bartowski has 18 at Q6_K and 18 at Q4_K. That accounts for 197,591,040 bytes of the gap; the remaining 768 is header and alignment.

Neither is wrong. But "Q4_K_M" is a recipe name, not a specification, and two files carrying it can differ by 188 MiB, which is enough to be the difference between fitting and spilling on an 8 GB card. If a model that "should" fit doesn't, check the actual file size rather than the label.

Where to go next

Run the same model at two quant levels and compare on your own prompts rather than on benchmarks. Most people find the drop from Q6_K to Q4_K_M smaller than expected and the drop below Q4 larger. Then work out your real KV cache cost, since that's what the generic calculators get most wrong and what actually caps your usable context.

Before you consider buying anything: generation speed tracks memory bandwidth, not teraflops, which is why a used 3090 at 936 GB/s still outruns a lot of newer silicon. And two GPUs in llama.cpp's default layer-split mode do not add bandwidth. Two 3090s give you 48 GB of capacity at roughly one card's token rate, because the split is sequential. Capacity and speed are separate purchases.