GGUF vs AWQ vs FP8 vs NVFP4: which format for local LLMs?
Quick answer
GGUF vs AWQ vs FP8 vs NVFP4 — which should I use?
Match the format to the engine: GGUF for llama.cpp / Ollama / LM Studio; AWQ or FP8 for vLLM / TensorRT-LLM on NVIDIA; NVFP4 on Blackwell (RTX 50-series, PRO 6000, GB10) when the engine supports it. Quality at ~4-bit is usually close across well-tuned quants; speed differences come from kernels and whether speculative decoding is on.
Last updated:
Formats are how weights are stored and which engines can load them. Below is a practical map, plus RigBase rows where the same model was measured in more than one format.
What each format is for
| Format | Typical bits | Best with | Notes |
|---|---|---|---|
| GGUF (Q4_K_M, IQ4_XS…) | ~2–8 (Q4_K_M = 4.89) | llama.cpp, Ollama, LM Studio, koboldcpp | Bits from the llama.cpp quantize table. Widest CPU/GPU/Apple coverage. |
| AWQ | ~4 | vLLM, Hugging Face CUDA stacks | Activation-aware weight quant; common for Tensor Parallel multi-GPU. |
| FP8 | 8 | vLLM, TensorRT-LLM on Ada/Hopper/Blackwell | Higher fidelity than 4-bit; needs more VRAM. |
| NVFP4 | ~4.5 (block FP8 scales) | Blackwell GPUs (50-series, PRO 6000, GB10) | Per NVIDIA’s NVFP4 post. |
| MXFP4 | ~4.25 | gpt-oss official releases, some MLX/llama.cpp paths | Used on gpt-oss 20B/120B native-ish packs in RigBase. |
Same models, different formats (measured)
| Model | Hardware | Quant / format | Engine | tok/s |
|---|---|---|---|---|
| Qwen3.6 35B-A3B | 4x RTX 3090 | IQ4_XS (1 GPU) | llama.cpp | 115-133 |
| Qwen3.6 35B-A3B | 4x RTX 3090 | AWQ-INT4 (TP=2) | vLLM | 149 |
| Qwen3.6 35B-A3B | 4x RTX 3090 | AWQ-INT4 + MTP-3 (TP=2) | vLLM | 225-231 ⚡ |
| Flint 40B-A2.5B | RTX 4090 | NVFP4 | llama.cpp 0.7.0-dev (build 118 | 179.4 |
| Flint 40B-A2.5B | RTX 4090 | UD-Q3_K_XL | llama.cpp 0.7.0-dev (build 118 | 169.2 |
| Flint 40B-A2.5B | RTX 5090 32GB | NVFP4 GGUF (RecursiveSI/flint-nvfp4-gguf), f1 | llama.cpp b11803, no speculati | 224.9 |
| Flint 40B-A2.5B | RTX 5090 32GB | NVFP4 GGUF, f16 KV | llama.cpp b11803, no speculati | 110.2 |
Do not crown a format from one row: engine, tensor parallel, context and speculative decoding dominate. Use the table to see what actually runs.
Related: VRAM calculator (GGUF + NVFP4 + vLLM-style) · multi-GPU 3090 · DGX Spark NVFP4 recipes
FAQ
GGUF vs AWQ vs FP8 vs NVFP4 — which should I use?
Match the format to the engine: GGUF for llama.cpp / Ollama / LM Studio; AWQ or FP8 for vLLM / TensorRT-LLM on NVIDIA; NVFP4 on Blackwell (RTX 50-series, PRO 6000, GB10) when the engine supports it. Quality at ~4-bit is usually close across well-tuned quants; speed differences come from kernels and whether speculative decoding is on.
What is NVFP4?
NVIDIA’s 4-bit floating format: about 4.5 bits per weight because every 16 values share an FP8 scale (NVIDIA developer blog on NVFP4). It needs Blackwell-class hardware and a supporting engine (vLLM, TensorRT-LLM, some llama.cpp builds).
Is AWQ better than GGUF Q4?
Not universally. On the same 4×3090 lab, Qwen3.6 35B-A3B measured 115–133 tok/s as IQ4_XS in llama.cpp (1 GPU) and 149 tok/s as AWQ-INT4 with vLLM TP=2 — different engines and sharding, so it is not a pure format contest. Prefer the stack you can run, then pick a ≥Q4-class quant.