Models / Nemotron 3.5 Bolt E4B
Best home rig for Nemotron 3.5 Bolt E4B
Quick answer
How much VRAM does Nemotron 3.5 Bolt E4B need?
The weights are about 4.9 GB at Q4_K_M (8B params × 4.89 bits ÷ 8, before context/KV cache). Add memory for context (KV cache) and the runtime. The smallest-memory setup with a measured result here, fully in memory, is Apple M4 (16 GB) at 28.9 tok/s.
What is the fastest measured hardware for Nemotron 3.5 Bolt E4B?
Strix Halo (Radeon 8060S): 96.4 tok/s generation (strix-halo-engine-lab, profile quality, EXL3 M4 4.65 bpw (hyukudan) + MTP drafter), from https://huggingface.co/hyukudan/Nemotron-3.5-Bolt-E4B-EXL3.
Last updated:
Legacy model8B dense8 measured resultsfastest plain decode: 73.5 tok/s model info ↗
Older model. For current picks see current models.
Real, measured generation speeds for Nemotron 3.5 Bolt E4B on home hardware, fastest first. Different quants/engines aren't directly comparable.
Prompt processing = prefill speed reported by the source, @ prompt length. * est. TTFT @8k is calculated from measured prefill, not measured. Why it matters.
Quality cost of each quant for Nemotron 3.5 Bolt E4B
How far each quantization drifts from the full-precision (BF16) model, as published by the quant maker. Lower KL divergence (KLD) is closer to the original; same-top-token % is how often the quant picks the same next token as BF16. Q4_K_M is usually the sweet spot; below Q4 the drift grows fast, which is why RigBase only lists Q4-or-higher speed results.
| Quant | File size | Mean KLD vs BF16 | Same top token |
|---|---|---|---|
| EXL3 M4 4.65 bpw (hyukudan) | 9.71 GB | 0.0367 | 92.94% |
| Q4_K_M (NVIDIA official 4GB GGUF) | 6.18 GB | 0.141 | 87.0% |
Source: huggingface.co ↗. hyukudan (EXL3 quant publisher) model card: measured through strix-halo-engine-lab's decode path on a held-out 40-document, 8-language set against the official BF16 logits. The EXL3 download is larger because it keeps the embedding tables in BF16; per token it reads 2.70 GB vs 3.13 GB for Q4_K_M. KLD from different publishers, datasets or llama.cpp builds isn't directly comparable.
Memory math: how much VRAM / RAM does Nemotron 3.5 Bolt E4B need?
Nemotron 3.5 Bolt E4B has 8B parameters (source: NVIDIA: 8B total, 4B effective (post-trained from Gemma 4 E4B); llama.cpp reports 7.52B). Weight size = parameters × bits-per-weight ÷ 8, before context (KV cache) and runtime overhead.
| Format | Bits/weight | Weights | Smallest common memory size that holds the weights alone |
|---|---|---|---|
| Q8_0 | 8.50 | 8.5 GB | 12 GB |
| Q6_K | 6.56 | 6.6 GB | 8 GB |
| Q5_K_M | 5.70 | 5.7 GB | 8 GB |
| Q4_K_M | 4.89 | 4.9 GB | 8 GB |
| Q3_K_M | 4.00 | 4.0 GB | 8 GB |
| Q2_K | 3.16 | 3.2 GB | 8 GB |
| As measured: EXL3 M4 4.65 bpw (hyukudan, 9.71 GB download, 2.70 GB read p | 4.65 | ≈4.7 GB | 8 GB |
| As measured: EXL3 M4 4.65 bpw (hyukudan) + MTP drafter | 4.65 | ≈4.7 GB | 8 GB |
Bits-per-weight for GGUF quants are llama.cpp's own figures (llama.cpp quantize README). "As measured" rows apply the source's stated bits-per-weight to the parameter count; real files differ slightly because some layers stay at higher precision. Try other combinations in the VRAM calculator.
FAQ
How much VRAM does Nemotron 3.5 Bolt E4B need?
The weights are about 4.9 GB at Q4_K_M (8B params × 4.89 bits ÷ 8, before context/KV cache). Add memory for context (KV cache) and the runtime. The smallest-memory setup with a measured result here, fully in memory, is Apple M4 (16 GB) at 28.9 tok/s.
What is the fastest measured hardware for Nemotron 3.5 Bolt E4B?
Strix Halo (Radeon 8060S): 96.4 tok/s generation (strix-halo-engine-lab, profile quality, EXL3 M4 4.65 bpw (hyukudan) + MTP drafter), from https://huggingface.co/hyukudan/Nemotron-3.5-Bolt-E4B-EXL3.
Related: what GPU do you need for 2026 models · all builds
Speed with and without MTP / DFlash on the same machines: measured speed-ups.