It loaded, but generation is painfully slow

This is the failure mode that doesn't announce itself. When a model doesn't fit, most runtimes will happily place the remainder in system memory and run anyway. System RAM bandwidth is roughly an order of magnitude below GPU memory bandwidth, and since token generation is bandwidth-bound, the offloaded fraction dominates your token rate. People conclude their GPU is slow when the real problem is that a few layers never made it onto it.

1. 1
   Confirm it actually fit

   Check how many layers were offloaded to the GPU and compare reported VRAM use against your card. If the layer count is below the model's total, that's your answer.

2. 2
   Make it fit, using the same ladder as an out-of-memory error

   Shorter context, quantized KV cache, one quantization level down. Anything that gets the whole model resident is worth more than any other tuning.

3. 3
   If it cannot fit, prefer a mixture-of-experts model

   MoE models read only their routed experts per token, so the penalty for keeping experts in system RAM is much smaller than for a dense model of equivalent size.

Prompt processing being slow while generation is fine is a different problem and usually normal: prefill is compute-bound rather than bandwidth-bound, and quantizing weights does not speed it up.

Rather than guessing, pick your model and card and read the grid: every quantization at every context, with the memory each combination actually needs.