More
Submit build

Models / Qwen3.5 397B-A17B

Best home rig for Qwen3.5 397B-A17B

Quick answer

How much VRAM does Qwen3.5 397B-A17B need?
The weights are about 246.6 GB at Q4_K_M (403B params × 4.89 bits ÷ 8, before context/KV cache). Add memory for context (KV cache) and the runtime. The smallest-memory setup with a measured result here, fully in memory, is Apple M5 Ultra (256 GB) at 61.6 tok/s.

What is the fastest measured hardware for Qwen3.5 397B-A17B?
Apple M5 Ultra: 61.6 tok/s generation (mlx-lm 0.32.2 (MLX 0.32.4), 4-bit MLX (mlx-community, ~224 GB)), from https://bohemiansilicon.com/blog/qwen3-5-397b-mac-studio-m5-ultra-256gb/.

Last updated:

Legacy model403B total · 17B active1 measured resultfastest plain decode: 61.6 tok/s model info ↗

Newer model available: Qwen3.8 Flash-Next. Qwen3.5 397B-A17B is a legacy model; its numbers stay here for reference. Qwen’s current large open MoE. source ↗

Real, measured generation speeds for Qwen3.5 397B-A17B on home hardware, fastest first. Different quants/engines aren't directly comparable.

BuildQuantEnginetok/sPrompt processing (tok/s)CostSource
Mac Studio M5 Ultra 256 GB: Qwen3.5 397B-A17B 4-bit MLX (224 GB) at 61.6 tok/s, 44.1 at 131K context (Bohemian Silicon)4-bit MLX (mlx-community, ~224 GB)mlx-lm 0.32.2 (MLX 0.32.4)61.61,265 @8K · est. TTFT @8k ≈6.5 s*bohemiansilicon.com ↗

Prompt processing = prefill speed reported by the source, @ prompt length. * est. TTFT @8k is calculated from measured prefill, not measured. Why it matters.

Rent before you buy: try this model on a cloud GPU for a few dollars first: Runpod (referral link; sign up with Google to get the new-user bonus)

Memory math: how much VRAM / RAM does Qwen3.5 397B-A17B need?

Qwen3.5 397B-A17B has 403B parameters, about 17B active per token (source: Qwen Hugging Face repo (Feb 2026): 397B total / ~17B active MoE; HF API lists 403.4B params in the safetensors (vision tower included)). Weight size = parameters × bits-per-weight ÷ 8, before context (KV cache) and runtime overhead.

FormatBits/weightWeightsSmallest common memory size that holds the weights alone
Q8_08.50428.2 GB512 GB
Q6_K6.56330.6 GB512 GB
Q5_K_M5.70287.3 GB512 GB
Q4_K_M4.89246.6 GB256 GB
Q3_K_M4.00201.3 GB256 GB
Q2_K3.16159.1 GB192 GB

Bits-per-weight for GGUF quants are llama.cpp's own figures (llama.cpp quantize README). "As measured" rows apply the source's stated bits-per-weight to the parameter count; real files differ slightly because some layers stay at higher precision. Try other combinations in the VRAM calculator.

Why it can still run on smaller GPUs: it is a mixture-of-experts model, so only about 17B of its 403B parameters are used for each generated token. Runtimes such as llama.cpp (--n-cpu-moe), ExLlamaV3 or Project Maya can keep most expert weights in system RAM and only the shared layers and hot experts in VRAM. Then you need VRAM + system RAM ≥ weights, and speed is limited mainly by RAM bandwidth.

FAQ

How much VRAM does Qwen3.5 397B-A17B need?

The weights are about 246.6 GB at Q4_K_M (403B params × 4.89 bits ÷ 8, before context/KV cache). Add memory for context (KV cache) and the runtime. The smallest-memory setup with a measured result here, fully in memory, is Apple M5 Ultra (256 GB) at 61.6 tok/s.

What is the fastest measured hardware for Qwen3.5 397B-A17B?

Apple M5 Ultra: 61.6 tok/s generation (mlx-lm 0.32.2 (MLX 0.32.4), 4-bit MLX (mlx-community, ~224 GB)), from https://bohemiansilicon.com/blog/qwen3-5-397b-mac-studio-m5-ultra-256gb/.

Related: what GPU do you need for 2026 models · all builds