Models / Inkling 2 Mini
Best home rig for Inkling 2 Mini
Quick answer
How much VRAM does Inkling 2 Mini need?
The weights are about 20 GB in its native format (Thinking Machines Lab model card: 26.4B total / 3.4B active looped MoE (middle layers run 3 times, ~5.4B effective active), text + image in, up to 1M context, Apache 2.0, released 20 Nov 2026; one ~20 GB NVFP4/MXFP8 checkpoint). Add memory for context (KV cache) and the runtime. The smallest-memory setup with a measured result here, fully in memory, is RTX 3090 (24 GB) at 73.5 tok/s.
What is the fastest measured hardware for Inkling 2 Mini?
2x RTX 3090: 136.6 tok/s generation (vLLM TP2 + Ampere overlay, NVLink, 1 stream, W4A16 INT4 + DSpark), from https://github.com/tonyd2wild/Inkling-2-Mini-W4A16-RTX3090.
Last updated:
Current model26.4B total · 3.4B active31 measured resultsfastest plain decode: 59.7 tok/s model info ↗
Will it fit my GPU? Best value GPU per euro →
Real, measured generation speeds for Inkling 2 Mini on home hardware, fastest first. Different quants/engines aren't directly comparable.
Prompt processing = prefill speed reported by the source, @ prompt length. * est. TTFT @8k is calculated from measured prefill, not measured. Why it matters.
Quality cost of each quant for Inkling 2 Mini
How far each quantization drifts from the full-precision (BF16) model, as published by the quant maker. Lower KL divergence (KLD) is closer to the original; same-top-token % is how often the quant picks the same next token as BF16. Q4_K_M is usually the sweet spot; below Q4 the drift grows fast, which is why RigBase only lists Q4-or-higher speed results.
| Quant | File size | Mean KLD vs BF16 | Same top token |
|---|---|---|---|
| UD-Q4_K_XL | 17.63 GB | 0.0107 | 98.0% |
| UD-Q4_K_M | 16.61 GB | 0.0108 | 97.9% |
| UD-IQ4_XS | 15.73 GB | 0.0109 | 97.7% |
| UD-Q3_K_XL below Q4 | 13.63 GB | 0.0174 | 96.3% |
| UD-IQ3_XXS below Q4 | 11.66 GB | 0.0269 | 94.7% |
| UD-Q2_K_XL below Q4 | 9.14 GB | 0.0595 | 91.6% |
| UD-IQ2_XXS below Q4 | 8.49 GB | 0.0726 | 90.6% |
| UD-IQ1_M below Q4 | 8.02 GB | 0.0867 | 90.2% |
| UD-IQ1_S below Q4 | 7.76 GB | 0.0911 | 89.4% |
Source: unsloth.ai ↗. Unsloth (quant publisher) run-locally guide, Nov 2026: each GGUF vs its UD-Q8_K_XL file (22.87 GB) across seven domains with identical llama.cpp settings. The reference keeps the official NVFP4 expert weights and stores the rest in BF16, i.e. the released precision, so it is the practical full-precision baseline for this model. KLD from different publishers, datasets or llama.cpp builds isn't directly comparable.
Memory math: how much VRAM / RAM does Inkling 2 Mini need?
Inkling 2 Mini has 26.4B parameters, about 3.4B active per token (source: Thinking Machines Lab model card: 26.4B total / 3.4B active looped MoE (middle layers run 3 times, ~5.4B effective active), text + image in, up to 1M context, Apache 2.0, released 20 Nov 2026; one ~20 GB NVFP4/MXFP8 checkpoint). Weight size = parameters × bits-per-weight ÷ 8, before context (KV cache) and runtime overhead.
Official checkpoint is already NVFP4/MXFP8 (~20 GB on disk); there is no separate BF16 download.
| Format | Bits/weight | Weights | Smallest common memory size that holds the weights alone |
|---|---|---|---|
| Native release format | – | ≈20 GB | 24 GB |
| Q8_0 | 8.50 | 28.1 GB | 32 GB |
| Q6_K | 6.56 | 21.7 GB | 24 GB |
| Q5_K_M | 5.70 | 18.8 GB | 24 GB |
| Q4_K_M | 4.89 | 16.2 GB | 24 GB |
| Q3_K_M | 4.00 | 13.2 GB | 16 GB |
| Q2_K | 3.16 | 10.4 GB | 12 GB |
| As measured: XQ mixed (~4.0 bpw routed experts, 13 of 24 layers keep offi | 4.00 | ≈13.2 GB | 16 GB |
Bits-per-weight for GGUF quants are llama.cpp's own figures (llama.cpp quantize README). "As measured" rows apply the source's stated bits-per-weight to the parameter count; real files differ slightly because some layers stay at higher precision. Try other combinations in the VRAM calculator.
Why it can still run on smaller GPUs: it is a mixture-of-experts model, so only about 3.4B of its 26.4B parameters are used for each generated token. Runtimes such as llama.cpp (--n-cpu-moe), ExLlamaV3 or Project Maya can keep most expert weights in system RAM and only the shared layers and hot experts in VRAM. Then you need VRAM + system RAM ≥ weights, and speed is limited mainly by RAM bandwidth.
Measured on setups with less GPU memory than the Q4_K_M / native weights (so part of the model lived in system RAM or was streamed from disk):
- AMD Radeon RX 9070 XT 16GB (16 GB) + 64 GB: 20.8 tok/s (IQ3_XXS (InterpreticAI, 10.9 GiB))
FAQ
How much VRAM does Inkling 2 Mini need?
The weights are about 20 GB in its native format (Thinking Machines Lab model card: 26.4B total / 3.4B active looped MoE (middle layers run 3 times, ~5.4B effective active), text + image in, up to 1M context, Apache 2.0, released 20 Nov 2026; one ~20 GB NVFP4/MXFP8 checkpoint). Add memory for context (KV cache) and the runtime. The smallest-memory setup with a measured result here, fully in memory, is RTX 3090 (24 GB) at 73.5 tok/s.
What is the fastest measured hardware for Inkling 2 Mini?
2x RTX 3090: 136.6 tok/s generation (vLLM TP2 + Ampere overlay, NVLink, 1 stream, W4A16 INT4 + DSpark), from https://github.com/tonyd2wild/Inkling-2-Mini-W4A16-RTX3090.
Which GPUs have measured results for Inkling 2 Mini?
2x RTX 3090, Apple M3 Ultra, Apple M5 Ultra, DGX Spark (GB10), RTX 3090, RTX 4090.
Can it run on… 2x RTX 3090 · Apple M3 Ultra · Apple M5 Ultra · DGX Spark (GB10) · RTX 3090 · RTX 4090
Related: what GPU do you need for 2026 models · all builds