Quantization for local LLMs: is Q4 good enough?
Quick answer
Is Q4 good enough for local LLMs?
For most chat and coding use, yes — Q4_K_M (≈4.89 bits/weight in llama.cpp) is the default sweet spot between quality and VRAM. RigBase’s calculator and guides treat Q4 as the baseline fit estimate. Going below Q4 (Q3, Q2, heavy IQ2) often saves VRAM but can hurt instruction-following and long-context coherence; we do not recommend it as a first choice.
Last updated:
Quantization shrinks weights so models fit on home GPUs. Q4 is the usual starting point; this page explains why RigBase advises not going below it for everyday use.
Approximate weight size (GB) by quant
| Model | Q8_0 | Q5_K_M | Q4_K_M | Q3_K_M |
|---|---|---|---|---|
| Qwen3.8 27B | 28.7 | 19.2 | 16.5 | 13.5 |
| Qwen3.6 35B-A3B | 37.7 | 25.3 | 21.7 | 17.7 |
| Gemma 4 31B | 32.6 | 21.9 | 18.8 | 15.3 |
| gpt-oss 20B | 22.2 | 14.9 | 12.8 | 10.4 |
GB ≈ params × bits ÷ 8. Bits from the llama.cpp quantize README. Native-release packs (e.g. gpt-oss MXFP4) may already be ~4-bit — check each VRAM page.
Practical rules
- Default: Q4_K_M (GGUF) or AWQ/NVFP4 ~4-bit on vLLM/Blackwell.
- Have spare VRAM? Try Q5_K_M or Q6_K before buying another GPU.
- Below Q4: only to squeeze a model that otherwise will not load; expect more refusals/nonsense on hard prompts.
- KV cache: quantising KV (q8_0 / FP8 cache) saves context memory with a different tradeoff than weight quant — see the calculator’s KV section.
More: format guide · Can I run it? · VRAM needed per model
FAQ
Is Q4 good enough for local LLMs?
For most chat and coding use, yes — Q4_K_M (≈4.89 bits/weight in llama.cpp) is the default sweet spot between quality and VRAM. RigBase’s calculator and guides treat Q4 as the baseline fit estimate. Going below Q4 (Q3, Q2, heavy IQ2) often saves VRAM but can hurt instruction-following and long-context coherence; we do not recommend it as a first choice.
Q4 vs Q5 vs Q8 — when to go higher?
Q5/Q6/Q8 help when you have spare VRAM and notice quality issues at Q4 (subtle reasoning, multilingual, creative prose). The weight cost scales roughly with bits: Q8_0 is ~8.5 bits/weight vs Q4_K_M ~4.89 (llama.cpp table).
What about NVFP4 / AWQ 4-bit?
They are also ~4-bit class formats aimed at similar quality/size tradeoffs on different engines. See GGUF vs AWQ vs FP8 vs NVFP4.
Diese Seite auf Deutsch: Ist Q4 gut genug?