Models / Mellum 2.1 12B-A2.5B

Best home rig for Mellum 2.1 12B-A2.5B

Current model12B total · 2.5B active1 measured resultfastest plain decode: 177.5 tok/s model info ↗

Will it fit my GPU? Best value GPU per euro →

Real, measured generation speeds for Mellum 2.1 12B-A2.5B on home hardware, fastest first. Different quants/engines aren't directly comparable.

Build	Quant	Engine	tok/s	Cost	Source
RTX 4070 12 GB running Mellum 2.1 12B-A2.5B Thinking Q4_K_M	Thinking GGUF Q4_K_M	llama.cpp	177.5		llm-bench.io ↗

Rent before you buy: try this model on a cloud GPU for a few dollars first: Runpod (referral link; sign up with Google to get the new-user bonus)

Memory math: how much VRAM / RAM does Mellum 2.1 12B-A2.5B need?

Mellum 2.1 12B-A2.5B has 12B parameters, about 2.5B active per token (source: 12B total / 2.5B active per the model name (JetBrains Mellum 2.1)). Weight size = parameters × bits-per-weight ÷ 8, before context (KV cache) and runtime overhead.

Format	Bits/weight	Weights	Smallest common memory size that holds the weights alone
Q8_0	8.50	12.8 GB	16 GB
Q6_K	6.56	9.8 GB	12 GB
Q5_K_M	5.70	8.6 GB	12 GB
Q4_K_M	4.89	7.3 GB	8 GB
Q3_K_M	4.00	6.0 GB	8 GB
Q2_K	3.16	4.7 GB	8 GB

Bits-per-weight for GGUF quants are llama.cpp's own figures (llama.cpp quantize README). "As measured" rows apply the source's stated bits-per-weight to the parameter count; real files differ slightly because some layers stay at higher precision. Try other combinations in the VRAM calculator.

Why it can still run on smaller GPUs: it is a mixture-of-experts model, so only about 2.5B of its 12B parameters are used for each generated token. Runtimes such as llama.cpp (--n-cpu-moe), ExLlamaV3 or Project Maya can keep most expert weights in system RAM and only the shared layers and hot experts in VRAM. Then you need VRAM + system RAM ≥ weights, and speed is limited mainly by RAM bandwidth.

FAQ

How much VRAM does Mellum 2.1 12B-A2.5B need?

The weights are about 7.3 GB at Q4_K_M (12B params × 4.89 bits ÷ 8, before context/KV cache). Add memory for context (KV cache) and the runtime.

What is the fastest measured hardware for Mellum 2.1 12B-A2.5B?

RTX 4070 12 GB running Mellum 2.1 12B-A2.5B Thinking Q4_K_M: 177.5 tok/s generation (llama.cpp, Thinking GGUF Q4_K_M), from https://llm-bench.io/benchmarks/cmv76txyv004p01m4h40td1d6.