Local Embeddings and Rerankers: The Model Is Never the Bottleneck

Start here

Download Qwen3-Embedding-0.6B — 1,191,586,416 bytes as bf16 safetensors, 639,150,592 as the official Q8_0 GGUF — and stop shopping. If that feels heavy, ibm-granite/granite-embedding-small-english-r2 is 95,332,048 bytes and handles 8192-token chunks. Retrieval is the one local-AI category where the model genuinely isn't the constraint. No GPU is required. The gap between "smallest thing that works" and "best open weights available" is about 15 GB of download and a few points of benchmark score.

What will cost you is the vector store, the reranker pass at query time, and the VRAM you take away from the generation model sharing the card.

Generation intuitions do not transfer

An embedding model runs one forward pass over your text and pools the hidden states into a single fixed-length vector. There is no decode loop. Everything you know about local generation — bandwidth-bound token rates, quantization making decode roughly 2.3x faster, KV cache growing with context — describes the autoregressive loop, and an embedder never enters it.

Embedding is pure prefill. That makes it compute-bound rather than bandwidth-bound, which has three consequences worth internalizing. Quantizing an embedder saves disk and RAM but buys little speed. Batch size matters enormously and per-item latency barely matters when you're indexing. And a CPU that produces an embarrassing 6 tok/s on an 8B chat model will chew through a document corpus at a perfectly reasonable rate, because it never pays the per-token round trip.

MTEB's memory field is a trap

MTEB is the right quality reference — it is the only benchmark with enough task coverage that gaming one dataset doesn't move the average. Its reported memory is a different story.

From mteb/models/model_meta.py on main, calculate_memory_usage_mb() documents its default as "estimate from n_parameters assuming FP32 (4 bytes per parameter)," and the fallback path is literally model_memory_bytes = n_parameters * 4. Only when called with fetch_from_hf=True does it read the real safetensors dtype map. Both kinds of value are sitting in the registry right now. Of the 694 ModelMeta entries I could parse out of mteb/models/model_implementations/ this session, 432 sit at exactly n_parameters × 4. That is correct for a checkpoint that ships FP32 and exactly double for one that ships BF16 or FP16.

I took the 275 text entries in that group with max_tokens >= 256, pulled the real safetensors dtype for 238 of them, and found 26 overstated by 2x:

The clearest tell: Qwen/Qwen3-Embedding-0.6B and IEITYuan/Yuan-embedding-2.0-en have identical n_parameters (595,776,512) and near-identical BF16 checkpoints of ~1.19 GB, and MTEB lists 1136 MB for one and 2272 MB for the other. Read the file size on the HuggingFace file listing instead. It is right there and it is not a guess.

Dimension is a storage decision, not a quality decision

Parameter count barely matters here. Output dimension does, because it multiplies by every chunk you will ever index. One million chunks costs 1.431 GiB at 384 dimensions, 2.861 GiB at 768, 3.815 GiB at 1024, and 15.259 GiB at Qwen3-Embedding-8B's 4096 — all in fp32, before any index structure. That last number is larger than the model.

Matryoshka-trained models let you truncate the vector, and Nomic publishes the actual tradeoff for nomic-embed-text-v1.5: MTEB 62.28 at 768 dimensions, 61.96 at 512, 61.04 at 256, 59.34 at 128, 56.10 at 64. Going 768 → 256 is a 3x storage cut for 1.24 points. Qwen3-Embedding supports user-defined dimensions from 32 up to its native 1024 / 2560 / 4096; EmbeddingGemma is 768 native with the same property. If your corpus is under ~100k chunks none of this matters and you should keep full dimension. Past a million, truncate first and reach for int8 or binary quantization second — those are 4x and 32x respectively, but how much recall they cost on your corpus is not something anyone has measured for you.

Max sequence length is where people actually get burned

sentence-transformers/all-MiniLM-L6-v2 has max_position_embeddings: 512 in config.json and max_seq_length: 256 in sentence_bert_config.json. Sentence-transformers honors the second one and silently truncates. If your chunks are 400 tokens, you are indexing the first 60% of each and wondering why retrieval is bad. multilingual-e5-large-instruct caps at 514. bge-m3 and the Granite r2 models do 8192. Check sentence_bert_config.json, not the marketing.

What to download

Every byte count below came from the HuggingFace API this session.

Two of these need a prefix or they underperform badly. Nomic requires search_document: on documents and search_query: on queries. Qwen3-Embedding wants queries wrapped as Instruct: <task>\nQuery: <text> and documents bare — its config_sentence_transformers.json ships the exact default string. Getting this wrong is the single most common cause of "the fancy model scored worse than MiniLM."

On quality, Qwen's published MTEB Multilingual table (leaderboard snapshot 2025-05-24) puts Qwen3-Embedding-0.6B at 64.33 mean, 4B at 69.45, 8B at 70.58, against bge-m3 at 59.56 and multilingual-e5-large-instruct at 63.22. Thirteen times the parameters from 0.6B to 8B buys 6.25 points. Start at 0.6B.

Rerankers are usually the better spend

A reranker is a cross-encoder: it reads the query and the document together in one pass and emits a relevance score. It cannot be precomputed, which is exactly why it's more accurate — nothing is compressed into a fixed vector. Retrieve 50 candidates with a cheap embedder, rerank them, keep 5. That pipeline routinely beats swapping in a 15 GB embedding model, and it costs one 400 MB download.

Budget the query-time cost honestly: 50 candidates at 512 tokens each is 25,600 tokens of prefill per query, versus ~20 tokens to embed the query. On CPU that is seconds, not milliseconds. bge-reranker-v2-m3 at Q4_K_M is 438,376,864 bytes and covers 100+ languages; Qwen3-Reranker-0.6B is 1,191,588,280 bytes bf16 and instruction-aware. Watch licenses — jina-reranker-v2-base-multilingual is CC-BY-NC-4.0, non-commercial, which most people miss.

Running it next to a generation model

They contend for the same VRAM, and a model that doesn't fit runs 5-20x slower once it spills. On a 24 GB card with a Q4_K_M 32B and a filled KV cache, you do not have 1.1 GiB spare. The fix is trivial: run the embedder and reranker on CPU in a second llama-server process. They're prefill-only, so the CPU penalty is a throughput hit during indexing and a fraction of a second at query time — not the catastrophe it would be for generation.

llama.cpp specifics

The standalone llama-embedding binary is gone from master; embeddings and reranking are served by llama-server.

llama-server -m qwen3-embedding-0.6b-q8_0.gguf --embedding -ub 8192 -b 8192
llama-server -m bge-reranker-v2-m3-Q4_K_M.gguf --reranking --pooling rank -ub 8192 -b 8192

The -ub flag is the one that bites. server.cpp states plainly that "embeddings require all tokens to be processed in a single ubatch," and clamps n_batch down to n_ubatch with a warning if you don't. Default -ub is 512, so an 8192-token chunk fails or gets silently cut. llama.cpp's own --embd-gemma-default preset sets n_ubatch = n_batch = 2048 and n_parallel = 32 for exactly this reason — 32 concurrent slots because throughput, not latency, is what indexing needs.