Builds / RTX Pro 6000 (Blackwell)
RTX PRO 6000 Blackwell 96 GB: Nemotron 3.5 Bolt 74B-A9B mixed NVFP4/FP8 on vLLM 0.32 at 121.7 tok/s for one user (150.9 with two cards TP=2), 4.0M-token KV cache
Last updated:
Source: docs.understand.tech ↗📏 How measured: reported by the original source; RigBase re-checked the numbers at that source
- GPU(s)
- RTX Pro 6000 (Blackwell)
Amazon · eBay (used) · Newegg - VRAM / unified memory (GB)
- 96
- CPU
- not stated
- Motherboard
- not stated
- RAM
- not stated
- PSU
- not stated
- Other
- vLLM 0.32.0, checkpoint primitive-ai/Nemotron-3.5-Bolt-74B-A9B-mixed-NVFP4-FP8 (46.7 GB), 44.0 GiB weights per GPU, KV cache 3,993,951 tokens, gpu-memory-utilization 0.92
- Reported total cost
- not stated
- Power
- not stated
Measured speed (generation tokens/sec)
| Model | Quant | Engine | tok/s | Prompt processing (tok/s) | Details |
|---|---|---|---|---|---|
| Nemotron 3.5 Bolt 74B-A9B | mixed NVFP4 + FP8 (primitive-ai) | vLLM 0.32.0, TP=1, b12x MoE backend | 121.7 | not stated | context: 262,144 ctx, chat workload, 1 session📏 Understand Tech serving test, one chat session, vLLM 0.32.0 (V2 model runner), b12x MoE backend, FlashInfer attention + Mamba backends, prefix caching on, 262,144 ctx; first token p95 0.6 s; per user 55.1 at 8 sessions, 24.0 at 32; chat throughput tops out near 859 tok/s at 64 sessions; repeats spread under 3% |
Hardware test by Understand Tech (AI-in-a-box documentation; page undated, vLLM 0.32.0). Same page, two cards with TP=2: 150.9 tok/s for one user (+24%), KV cache 13.3M tokens, about 856 tok/s chat ceiling; the authors recommend TP=1 per GPU because it serves 67-80% more chat sessions per GPU. The page also reports 107.6 tok/s for 1 request on the model card's own test vs 112.7 on the model card. CPU, RAM and board are not stated.
Source: https://docs.understand.tech/documentation/english-version/ai-in-a-box/hardware-experiments/nemotron-3.5-bolt-74b-a9b
Blank fields = not stated in the source. We never estimate build data; GPU street prices above are separately sourced and dated.
Rent before you buy: try this model on a cloud GPU for a few dollars first: Runpod (referral link; sign up with Google to get the new-user bonus)
Quick answers: Can RTX Pro 6000 (Blackwell, 96 GB) run Nemotron 3.5 Bolt 74B-A9B?