Source: https://getrigbase.com/guides/quantization-is-q4-good-enough/ Last updated: 2026-10-28 # Quantization for local LLMs: is Q4 good enough? Quick answer **Is Q4 good enough for local LLMs?** For most chat and coding use, yes — Q4_K_M (≈4.89 bits/weight in llama.cpp) is the default sweet spot between quality and VRAM. RigBase’s calculator and guides treat Q4 as the baseline fit estimate. Going below Q4 (Q3, Q2, heavy IQ2) often saves VRAM but can hurt instruction-following and long-context coherence; we do not recommend it as a first choice. Last updated: 28 Oct 2026 Quantization shrinks weights so models fit on home GPUs. Q4 is the usual starting point; this page explains why RigBase advises not going below it for everyday use. **Quick answer:Yes — start at Q4_K_M (or another solid ~4-bit format).** It halves VRAM vs Q8 with usually small quality loss for chat. Below Q4, treat it as an emergency fit for huge models, not the default. ## Approximate weight size (GB) by quant | Model | Q8_0 | Q5_K_M | Q4_K_M | Q3_K_M | |---|---|---|---|---| | [Qwen3.8 27B](https://getrigbase.com/models/qwen3-8-27b/) | 28.7 | 19.2 | 16.5 | 13.5 | | [Qwen3.6 35B-A3B](https://getrigbase.com/models/qwen3-6-35b-a3b/) | 37.7 | 25.3 | 21.7 | 17.7 | | [Gemma 4 31B](https://getrigbase.com/models/gemma-4-31b/) | 32.6 | 21.9 | 18.8 | 15.3 | | [gpt-oss 20B](https://getrigbase.com/models/gpt-oss-20b/) | 22.2 | 14.9 | 12.8 | 10.4 | GB ≈ params × bits ÷ 8. Bits from the [llama.cpp quantize README](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md). Native-release packs (e.g. gpt-oss MXFP4) may already be ~4-bit — check each [VRAM page](https://getrigbase.com/vram/). ## Practical rules - **Default:** Q4_K_M (GGUF) or AWQ/NVFP4 ~4-bit on vLLM/Blackwell. - **Have spare VRAM?** Try Q5_K_M or Q6_K before buying another GPU. - **Below Q4:** only to squeeze a model that otherwise will not load; expect more refusals/nonsense on hard prompts. - **KV cache:** quantising KV (q8_0 / FP8 cache) saves context memory with a different tradeoff than weight quant — see the [calculator’s KV section](https://getrigbase.com/can-it-run/). More: [format guide](https://getrigbase.com/guides/gguf-vs-awq-vs-fp8-vs-nvfp4/) · [Can I run it?](https://getrigbase.com/can-it-run/) · [VRAM needed per model](https://getrigbase.com/vram/) ## FAQ ### Is Q4 good enough for local LLMs? For most chat and coding use, yes — Q4_K_M (≈4.89 bits/weight in llama.cpp) is the default sweet spot between quality and VRAM. RigBase’s calculator and guides treat Q4 as the baseline fit estimate. Going below Q4 (Q3, Q2, heavy IQ2) often saves VRAM but can hurt instruction-following and long-context coherence; we do not recommend it as a first choice. ### Q4 vs Q5 vs Q8 — when to go higher? Q5/Q6/Q8 help when you have spare VRAM and notice quality issues at Q4 (subtle reasoning, multilingual, creative prose). The weight cost scales roughly with bits: Q8_0 is ~8.5 bits/weight vs Q4_K_M ~4.89 ([llama.cpp table](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md)). ### What about NVFP4 / AWQ 4-bit? They are also ~4-bit class formats aimed at similar quality/size tradeoffs on different engines. See [GGUF vs AWQ vs FP8 vs NVFP4](https://getrigbase.com/guides/gguf-vs-awq-vs-fp8-vs-nvfp4/). Ad · this space is available **Sponsor this spot: reach people building local AI rigs.** One clearly-labeled slot per page, no tracking scripts. [Sponsor this spot →](https://getrigbase.com/advertise/#inquiry) Diese Seite auf Deutsch: [Ist Q4 gut genug?](https://getrigbase.com/de/guides/quantisierung-ist-q4-gut-genug/)