More
Submit build

Models / Nemotron 3.5 Bolt E4B

Best home rig for Nemotron 3.5 Bolt E4B

Quick answer

How much VRAM does Nemotron 3.5 Bolt E4B need?
The weights are about 4.9 GB at Q4_K_M (8B params × 4.89 bits ÷ 8, before context/KV cache). Add memory for context (KV cache) and the runtime. The smallest-memory setup with a measured result here, fully in memory, is Apple M4 (16 GB) at 28.9 tok/s.

What is the fastest measured hardware for Nemotron 3.5 Bolt E4B?
Strix Halo (Radeon 8060S): 96.4 tok/s generation (strix-halo-engine-lab, profile quality, EXL3 M4 4.65 bpw (hyukudan) + MTP drafter), from https://huggingface.co/hyukudan/Nemotron-3.5-Bolt-E4B-EXL3.

Last updated:

Legacy model8B dense8 measured resultsfastest plain decode: 73.5 tok/s model info ↗

Older model. For current picks see current models.

Real, measured generation speeds for Nemotron 3.5 Bolt E4B on home hardware, fastest first. Different quants/engines aren't directly comparable.

BuildQuantEnginetok/sPrompt processing (tok/s)CostSource
Strix Halo (Ryzen AI Max+ 395, 128 GB): Nemotron 3.5 Bolt E4B at 55.5 tok/s on llama.cpp, 96.4 on strix-halo-engine-lab EXL3 with MTP drafterEXL3 M4 4.65 bpw (hyukudan) + MTP drafterstrix-halo-engine-lab, profile quality96.4 ⚡–huggingface.co ↗
Strix Halo (Ryzen AI Max+ 395, 128 GB): Nemotron 3.5 Bolt E4B at 55.5 tok/s on llama.cpp, 96.4 on strix-halo-engine-lab EXL3 with MTP drafterQ4_K_M (NVIDIA official 4GB GGUF, 6.18 GB) + MTP drafterstrix-halo-engine-lab, profile default91.9 ⚡–huggingface.co ↗
RTX 2060 Super 8 GB + Core i9-10900K: Qwen3.5 9B Q4_K_M at 54.4 tok/s, Gemma 4 12B at 27.7 (llama.cpp)Q4_K_M (NVIDIA official GGUF, 6.18 GB)llama.cpp b12064 server (CUDA 12.4, Windows build), -ngl 9973.5–mustafa.net ↗
Strix Halo (Ryzen AI Max+ 395, 128 GB): Nemotron 3.5 Bolt E4B at 55.5 tok/s on llama.cpp, 96.4 on strix-halo-engine-lab EXL3 with MTP drafterQ4_K_M (NVIDIA official 4GB GGUF, 6.18 GB)strix-halo-engine-lab, profile default55.8509 @8K · est. TTFT @8k ≈16.1 s*huggingface.co ↗
Strix Halo (Ryzen AI Max+ 395, 128 GB): Nemotron 3.5 Bolt E4B at 55.5 tok/s on llama.cpp, 96.4 on strix-halo-engine-lab EXL3 with MTP drafterQ4_K_M (NVIDIA official 4GB GGUF, 6.18 GB)llama.cpp 74a4fb0055.51,881 @8K · est. TTFT @8k ≈4.4 s*huggingface.co ↗
Strix Halo (Ryzen AI Max+ 395, 128 GB): Nemotron 3.5 Bolt E4B at 55.5 tok/s on llama.cpp, 96.4 on strix-halo-engine-lab EXL3 with MTP drafterEXL3 M4 4.65 bpw (hyukudan, 9.71 GB download, 2.70 GB read per token)strix-halo-engine-lab, profile quality50.2746 @8K · est. TTFT @8k ≈11.0 s*huggingface.co ↗
Mac mini M4 (16 GB): Qwen3.5 9B Q4_K_M at 17.7 tok/s, Gemma 4 12B at 13.1 (llama.cpp Metal)Q4_K_M (NVIDIA official GGUF, 6.18 GB)llama.cpp b11567 server (Metal)28.9–mustafa.net ↗
Intel NUC8i7BEH (Core i7-8559U, CPU only): Nemotron 3.5 Bolt E4B Q4_K_M at 5.0 tok/s, Qwen3.5 4B 5.63, Qwen3.5 9B 3.07 (llama.cpp)Q4_K_M (NVIDIA official GGUF, 6.18 GB)llama.cpp b11561 server (CPU)5.0–mustafa.net ↗

Prompt processing = prefill speed reported by the source, @ prompt length. * est. TTFT @8k is calculated from measured prefill, not measured. Why it matters.

Quality cost of each quant for Nemotron 3.5 Bolt E4B

How far each quantization drifts from the full-precision (BF16) model, as published by the quant maker. Lower KL divergence (KLD) is closer to the original; same-top-token % is how often the quant picks the same next token as BF16. Q4_K_M is usually the sweet spot; below Q4 the drift grows fast, which is why RigBase only lists Q4-or-higher speed results.

QuantFile sizeMean KLD vs BF16Same top token
EXL3 M4 4.65 bpw (hyukudan)9.71 GB0.036792.94%
Q4_K_M (NVIDIA official 4GB GGUF)6.18 GB0.14187.0%

Source: huggingface.co ↗. hyukudan (EXL3 quant publisher) model card: measured through strix-halo-engine-lab's decode path on a held-out 40-document, 8-language set against the official BF16 logits. The EXL3 download is larger because it keeps the embedding tables in BF16; per token it reads 2.70 GB vs 3.13 GB for Q4_K_M. KLD from different publishers, datasets or llama.cpp builds isn't directly comparable.

Rent before you buy: try this model on a cloud GPU for a few dollars first: Runpod (referral link; sign up with Google to get the new-user bonus)

Memory math: how much VRAM / RAM does Nemotron 3.5 Bolt E4B need?

Nemotron 3.5 Bolt E4B has 8B parameters (source: NVIDIA: 8B total, 4B effective (post-trained from Gemma 4 E4B); llama.cpp reports 7.52B). Weight size = parameters × bits-per-weight ÷ 8, before context (KV cache) and runtime overhead.

FormatBits/weightWeightsSmallest common memory size that holds the weights alone
Q8_08.508.5 GB12 GB
Q6_K6.566.6 GB8 GB
Q5_K_M5.705.7 GB8 GB
Q4_K_M4.894.9 GB8 GB
Q3_K_M4.004.0 GB8 GB
Q2_K3.163.2 GB8 GB
As measured: EXL3 M4 4.65 bpw (hyukudan, 9.71 GB download, 2.70 GB read p4.65≈4.7 GB8 GB
As measured: EXL3 M4 4.65 bpw (hyukudan) + MTP drafter4.65≈4.7 GB8 GB

Bits-per-weight for GGUF quants are llama.cpp's own figures (llama.cpp quantize README). "As measured" rows apply the source's stated bits-per-weight to the parameter count; real files differ slightly because some layers stay at higher precision. Try other combinations in the VRAM calculator.

FAQ

How much VRAM does Nemotron 3.5 Bolt E4B need?

The weights are about 4.9 GB at Q4_K_M (8B params × 4.89 bits ÷ 8, before context/KV cache). Add memory for context (KV cache) and the runtime. The smallest-memory setup with a measured result here, fully in memory, is Apple M4 (16 GB) at 28.9 tok/s.

What is the fastest measured hardware for Nemotron 3.5 Bolt E4B?

Strix Halo (Radeon 8060S): 96.4 tok/s generation (strix-halo-engine-lab, profile quality, EXL3 M4 4.65 bpw (hyukudan) + MTP drafter), from https://huggingface.co/hyukudan/Nemotron-3.5-Bolt-E4B-EXL3.

Related: what GPU do you need for 2026 models · all builds

Speed with and without MTP / DFlash on the same machines: measured speed-ups.