Models / Inkling 2 Mini / VRAM
How much VRAM do you need for Inkling 2 Mini?
Last updated:
Weight math from the model card, plus every measured RigBase result for Inkling 2 Mini. Leave extra room for context (KV cache).
Weight size by quant
| Quant | Bits/weight | Weights only |
|---|---|---|
| Native release format | – | ≈20 GB |
| Q8_0 | 8.50 | 28.1 GB |
| Q6_K | 6.56 | 21.7 GB |
| Q5_K_M | 5.70 | 18.8 GB |
| Q4_K_M | 4.89 | 16.2 GB |
| Q3_K_M | 4.00 | 13.2 GB |
| Q2_K | 3.16 | 10.4 GB |
Bits-per-weight for GGUF rows: llama.cpp quantize README. Parameter count: Thinking Machines Lab model card: 26.4B total / 3.4B active looped MoE (middle l. We advise staying at Q4 or above for chat quality — see Is Q4 good enough?.
Measured speeds for Inkling 2 Mini
| Hardware | Quant | Engine | tok/s |
|---|---|---|---|
| 2x RTX 3090 · 48 GB | W4A16 INT4 + DSpark | vLLM TP2 + Ampere overlay, NVLink, 1 str | 136.6 ⚡ |
| 2x RTX 3090 · 48 GB | W4A16 INT4 + DSpark | vLLM TP2 + Ampere overlay, NVLink, 1 str | 119 ⚡ |
| 2x RTX 3090 · 48 GB | W4A16 INT4 + DSpark | vLLM TP2 + Ampere overlay, NVLink P2P of | 100.5 ⚡ |
| 2x RTX 3090 · 48 GB | W4A16 INT4 | vLLM TP2 + Ampere overlay, 1 stream | 89.8 ⚡ |
| Apple M5 Ultra 80-core GPU (unified, Mac Studio) · 96 GB | oMLX oQ8e, Affine8 8-bit KV | oMLX v0.8.1, Lightning MTP | 83.4 ⚡ |
| Apple M5 Ultra 80-core GPU (unified, Mac Studio) · 96 GB | oMLX oQ8e, Affine8 8-bit KV | oMLX v0.8.1, Lightning MTP | 76.6 ⚡ |
| Apple M5 Ultra 80-core GPU (unified, Mac Studio) · 96 GB | oMLX oQ8e, Affine8 8-bit KV | oMLX v0.8.1, Lightning MTP | 74.7 ⚡ |
| RTX 3090 · 24 GB | W4A16 INT4 | vLLM TP1 + Ampere overlay, 1 stream | 73.5 ⚡ |
| RTX 4090 · 24 GB | XQ mixed (~4.0 bpw routed experts, 13 of 24 layers | Xolotl (xolotl-inkling) | 59.7 |
| RTX 4090 · 24 GB | Official precision, exact conversion (NVFP4 + MXFP | Xolotl (xolotl-inkling) | 57 |
| Apple M5 Ultra 80-core GPU (unified, Mac Studio) · 96 GB | oMLX oQ8e, Affine8 8-bit KV | oMLX v0.8.1, Lightning MTP | 56.2 ⚡ |
| Apple M5 Ultra 80-core GPU (unified, Mac Studio) · 96 GB | oMLX oQ8e, Affine8 8-bit KV | oMLX v0.8.1, Lightning MTP | 55.4 ⚡ |
⚡ = speculative decoding (MTP, DFlash…). Full list: best home rig for Inkling 2 Mini.
Related
All Inkling 2 Mini builds VRAM + KV calculator By VRAM size GGUF vs AWQ vs FP8 vs NVFP4 How much system RAM?
FAQ
How much VRAM do you need for Inkling 2 Mini?
Weights alone are about 20 GB in the native release format. Add memory for the KV cache (context) and the runtime. The smallest-memory measured result in RigBase is 16 GB (AMD Radeon RX 9070 XT 16GB) at 20.8 tok/s. Going below Q4 usually hurts quality for chat; see the Q4 guide.
What is the fastest measured speed for Inkling 2 Mini?
RTX 4090: 59.7 tok/s plain decode (XQ mixed (~4.0 bpw routed experts, 13 of 24 layers keep official NVFP4; Q8_0 backbone; 13.93 GB), abliterated, Xolotl (xolotl-inkling)).
Does system RAM replace VRAM?
Only for overflow. When the whole model fits in GPU or unified memory, system RAM barely matters for decode speed. See the system-RAM guide.
Referral link: if you sign up with Google and add $10, Runpod gives you and RigBase bonus credit (terms). It never affects our data.