Source: https://getrigbase.com/guides/gguf-vs-awq-vs-fp8-vs-nvfp4/ Last updated: 2026-10-28 # GGUF vs AWQ vs FP8 vs NVFP4: which format for local LLMs? Quick answer **GGUF vs AWQ vs FP8 vs NVFP4 — which should I use?** Match the format to the engine: GGUF for llama.cpp / Ollama / LM Studio; AWQ or FP8 for vLLM / TensorRT-LLM on NVIDIA; NVFP4 on Blackwell (RTX 50-series, PRO 6000, GB10) when the engine supports it. Quality at ~4-bit is usually close across well-tuned quants; speed differences come from kernels and whether speculative decoding is on. Last updated: 28 Oct 2026 Formats are how weights are stored and which engines can load them. Below is a practical map, plus RigBase rows where the same model was measured in more than one format. **Quick answer:** Use **GGUF** with llama.cpp-family tools, **AWQ / FP8** with vLLM on NVIDIA, and **NVFP4** on Blackwell when available. Stay around 4-bit or above for chat; see [Is Q4 good enough?](https://getrigbase.com/guides/quantization-is-q4-good-enough/). ## What each format is for | Format | Typical bits | Best with | Notes | |---|---|---|---| | **GGUF** (Q4_K_M, IQ4_XS…) | ~2–8 (Q4_K_M = 4.89) | llama.cpp, Ollama, LM Studio, koboldcpp | Bits from the [llama.cpp quantize table](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md). Widest CPU/GPU/Apple coverage. | | **AWQ** | ~4 | vLLM, Hugging Face CUDA stacks | Activation-aware weight quant; common for Tensor Parallel multi-GPU. | | **FP8** | 8 | vLLM, TensorRT-LLM on Ada/Hopper/Blackwell | Higher fidelity than 4-bit; needs more VRAM. | | **NVFP4** | ~4.5 (block FP8 scales) | Blackwell GPUs (50-series, PRO 6000, GB10) | Per [NVIDIA’s NVFP4 post](https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/). | | **MXFP4** | ~4.25 | gpt-oss official releases, some MLX/llama.cpp paths | Used on gpt-oss 20B/120B native-ish packs in RigBase. | ## Same models, different formats (measured) | Model | Hardware | Quant / format | Engine | tok/s | |---|---|---|---|---| | [Qwen3.6 35B-A3B](https://getrigbase.com/models/qwen3-6-35b-a3b/) | [4x RTX 3090](https://getrigbase.com/builds/4x3090-qwen36-vllm-tp2/) | IQ4_XS (1 GPU) | llama.cpp | **115-133** | | [Qwen3.6 35B-A3B](https://getrigbase.com/models/qwen3-6-35b-a3b/) | [4x RTX 3090](https://getrigbase.com/builds/4x3090-qwen36-vllm-tp2/) | AWQ-INT4 (TP=2) | vLLM | **149** | | [Qwen3.6 35B-A3B](https://getrigbase.com/models/qwen3-6-35b-a3b/) | [4x RTX 3090](https://getrigbase.com/builds/4x3090-qwen36-vllm-tp2/) | AWQ-INT4 + MTP-3 (TP=2) | vLLM | **225-231** ⚡ | | [Flint 40B-A2.5B](https://getrigbase.com/models/flint-40b-a2-5b/) | [RTX 4090](https://getrigbase.com/builds/llmspeed-rtx-4090-flint-40b-a2-5b-oct-2026/) | NVFP4 | llama.cpp 0.7.0-dev (build 118 | **179.4** | | [Flint 40B-A2.5B](https://getrigbase.com/models/flint-40b-a2-5b/) | [RTX 4090](https://getrigbase.com/builds/llmspeed-rtx-4090-flint-40b-a2-5b-oct-2026/) | UD-Q3_K_XL | llama.cpp 0.7.0-dev (build 118 | **169.2** | | [Flint 40B-A2.5B](https://getrigbase.com/models/flint-40b-a2-5b/) | [RTX 5090 32GB](https://getrigbase.com/builds/rtx-5090-flint-40b-nvfp4-llamacpp-niklaslenz/) | NVFP4 GGUF (RecursiveSI/flint-nvfp4-gguf), f1 | llama.cpp b11803, no speculati | **224.9** | | [Flint 40B-A2.5B](https://getrigbase.com/models/flint-40b-a2-5b/) | [RTX 5090 32GB](https://getrigbase.com/builds/rtx-5090-flint-40b-nvfp4-llamacpp-niklaslenz/) | NVFP4 GGUF, f16 KV | llama.cpp b11803, no speculati | **110.2** | Do not crown a format from one row: engine, tensor parallel, context and speculative decoding dominate. Use the table to see what actually runs. Related: [VRAM calculator (GGUF + NVFP4 + vLLM-style)](https://getrigbase.com/can-it-run/) · [multi-GPU 3090](https://getrigbase.com/guides/multi-gpu-rtx-3090-local-llm/) · [DGX Spark NVFP4 recipes](https://getrigbase.com/dgx-spark/) ## FAQ ### GGUF vs AWQ vs FP8 vs NVFP4 — which should I use? Match the format to the engine: GGUF for llama.cpp / Ollama / LM Studio; AWQ or FP8 for vLLM / TensorRT-LLM on NVIDIA; NVFP4 on Blackwell (RTX 50-series, PRO 6000, GB10) when the engine supports it. Quality at ~4-bit is usually close across well-tuned quants; speed differences come from kernels and whether speculative decoding is on. ### What is NVFP4? NVIDIA’s 4-bit floating format: about 4.5 bits per weight because every 16 values share an FP8 scale (NVIDIA developer blog on NVFP4). It needs Blackwell-class hardware and a supporting engine (vLLM, TensorRT-LLM, some llama.cpp builds). ### Is AWQ better than GGUF Q4? Not universally. On the same 4×3090 lab, Qwen3.6 35B-A3B measured 115–133 tok/s as IQ4_XS in llama.cpp (1 GPU) and 149 tok/s as AWQ-INT4 with vLLM TP=2 — different engines and sharding, so it is not a pure format contest. Prefer the stack you can run, then pick a ≥Q4-class quant. Ad · this space is available **Sponsor this spot: reach people building local AI rigs.** One clearly-labeled slot per page, no tracking scripts. [Sponsor this spot →](https://getrigbase.com/advertise/#inquiry)