Guides / Inkling 2 Mini
Can my rig run Inkling 2 Mini?
Quick answer
Can my rig run Inkling 2 Mini?
Often yes if you have about 24 GB of VRAM, or 32 GB or more of unified memory on Apple silicon. Thinking Machines' model card says the official checkpoint is about 20 GB (NVFP4/MXFP8) and recommends 32 GB or more on a Mac or a GPU. Unsloth's run-locally guide puts the recommended UD-Q4_K_XL GGUF at 17.6 GB and the lossless 8-bit file at 24 GB. A 16 GB card can run a smaller 3-bit GGUF: an RX 9070 XT measured 20.8 tok/s on an IQ3_XXS file (10.9 GiB) in day-one llama.cpp.
Last updated:
Inkling 2 Mini is Thinking Machines Lab's 20 November 2026 open-weights model: a looped mixture-of-experts transformer with 26.4B total parameters, 3.4B active, and about 5.4B effective active parameters per token because the middle 12 layers run three times. The official checkpoint is already NVFP4/MXFP8, about 20 GB, with up to 1M context, and it is released under Apache 2.0. This page collects every sourced local speed RigBase has, and separates vendor peak claims from independent measurements.
Hype vs measured
The model launched with a lot of attention. Thinking Machines' announcement (20 November 2026) and X post describe a looped transformer that is “small enough to run on your laptop,” scores 20.1 on the AA Index, and is much smaller than Inkling 2 Small (594B parameters). The lab also said researchers ran it offline on a MacBook Pro. Those statements are from the lab, not from RigBase measurements.
The same day, EXO Labs posted that they had worked with Thinking Machines to make Inkling 2 Mini run fast locally:
- 210 tok/s decode and 3,620 tok/s prefill on a 96 GB M5 Ultra Mac Studio
- 125 tok/s decode and 1,750 tok/s prefill on a 48 GB M5 Max MacBook Pro
- It “built a native weather app 1.8× faster than Claude Sonnet 5.5” on an M5 Ultra Mac Studio, and 11× cheaper at API pricing
A follow-up on the same thread said EXO got access on Wednesday, had it running at 30 tok/s in three hours, and at 210 tok/s a day later on an upcoming Apple Silicon inference engine, with architecture-specific kernel work and “better use of DSpark with adaptive drafting.” RigBase looked for a blog post, GitHub README or model card behind those figures on 21 Nov 2026 (blog.exolabs.net and benchmarks.exolabs.net had no Inkling 2 Mini write-up). The X posts do not state the quant, the context length, or a public software version. Treat 210 and 125 as vendor-reported peaks from EXO, with Thinking Machines, not independently measured.
Independent numbers on the same chip class are lower. A public oMLX session on a 96 GB M5 Ultra (oQ8e, Lightning MTP on) measured 83.4 tok/s at 4k context and 56.2 tok/s at 32k, with prefill of 3,225 and 2,968 tok/s. On a 64 GB M5 Pro Mac mini, pratikgx measured 29.95 tok/s in llama.cpp with the official IQ4_NL GGUF at a 512-token prompt, with no speculative decoding. Looping is the main reason a “3.4B-active” model is not as fast as a typical small MoE: each token runs the middle layers three times (~5.4B effective active parameters), as the model card explains.
Sourced speed by hardware
Generation (decode) tok/s is not comparable 1:1 across quants, engines or speculative decoding. Prefill is prompt-processing speed at the context the source measured. 31 measured results from 8 builds are in RigBase today; the table below shows the headline and long-context rows. Every number links to its build page and original source.
| Hardware | Decode tok/s | Prefill tok/s | Context | Quant | Engine | Kind | Source |
|---|---|---|---|---|---|---|---|
| Mac Studio M5 Ultra 96 GB (80-core) | 83.4 | 3,225 @4k | 4k tokens | oMLX oQ8e, Affine8 8-bit KV | oMLX v0.8.1, Lightning MTP | Independent | source |
| Mac Studio M5 Ultra 96 GB (80-core) | 56.2 | 2,968 @32k | 32k tokens | oMLX oQ8e, Affine8 8-bit KV | oMLX v0.8.1, Lightning MTP | Independent | source |
| RTX 4090 24 GB | 59.7 | 1,162 @8K | 8K prompt, 1,024 generated | XQ mixed (~4.0 bpw routed experts, 13 of 24 layers keep official NVFP4; Q8_0 bac | Xolotl (xolotl-inkling) | Independent | source |
| RTX 4090 24 GB | 57 | 1,131 @8K | 8K prompt, 1,024 generated | Official precision, exact conversion (NVFP4 + MXFP8 experts, BF16 rest; 22.9 GB) | Xolotl (xolotl-inkling) | Independent | source |
| 2x RTX 3090 24 GB, NVLink | 137 | – | 458,752 per request (TP2 max lane, text-only): BF16 KV, 483,850-token pool | W4A16 INT4 + DSpark | vLLM TP2 + Ampere overlay, NVLink, 1 stream | Independent | source |
| 2x RTX 3090 24 GB, NVLink | 119 | 2,302 @439,384 | 439,384-token needle prompt on the TP2 max lane (458,752 per request) | W4A16 INT4 + DSpark | vLLM TP2 + Ampere overlay, NVLink, 1 stream | Independent | source |
| 2x RTX 3090 24 GB (no spec decoding) | 89.8 | – | not stated (TP2 quality comparison, greedy) | W4A16 INT4 | vLLM TP2 + Ampere overlay, 1 stream | Independent | source |
| RTX 3090 24 GB (one card) | 73.5 | – | 8,192 default, 32,768 max (33,392-token BF16 KV pool at gpu-memory-utilization 0.95) | W4A16 INT4 | vLLM TP1 + Ampere overlay, 1 stream | Independent | source |
| DGX Spark (128 GB unified) | 47.7 | – | 262K standard tier: BF16 KV, gpu-memory-utilization 0.80 (68.89 GiB KV, 1,346,671-token pool) | NVFP4/MXFP8 + DSpark | vLLM TP1, 1 stream | Independent | source |
| DGX Spark (128 GB unified) | 47.2 | 1,693 @199,989 | 199,989-token prompt on the 1M max FP8 tier (74 GiB FP8 KV) | NVFP4/MXFP8 + DSpark, FP8 KV | vLLM TP1, 1 stream | Independent | source |
| DGX Spark (128 GB unified) | 18.4 | 598 @949,953 | 949,953-token prompt on the 1M max FP8 tier (74 GiB FP8 KV) | NVFP4/MXFP8 + DSpark, FP8 KV | vLLM TP1, 1 stream | Independent | source |
| Mac Studio M3 Ultra 512 GB (80-core) | 40.3 | 1,169 @1k | 1k tokens | oMLX Inkling-2-Mini (as labelled) | oMLX v0.8.1, MTP off | Independent | source |
| Mac mini M5 Pro 64 GB | 29.95 | 491 @512 | 512-token prompt, 256 generated | ggml-org IQ4_NL (17.63 GB; 4-bit experts in layers 0-19, Q8_0 elsewhere, loop co | llama.cpp PR #32694 (d7cb78c91, build 12587), Metal | Independent | source |
| Mac mini M5 Pro 64 GB | 17.39 | 95.29 @131,072 | 131,072-token prompt, 256 generated | ggml-org IQ4_NL (17.63 GB; 4-bit experts in layers 0-19, Q8_0 elsewhere, loop co | llama.cpp PR #32694 (d7cb78c91, build 12587), Metal | Independent | source |
| RX 9070 XT 16 GB (Windows) | 20.8 | 403 @2048 | llama-bench tg64 | IQ3_XXS (InterpreticAI, 10.9 GiB) | llama.cpp b12600 Vulkan (Windows) | Independent | source |
oMLX M5 Ultra rows use Lightning MTP; the 128k generation figure in that session (74.7 tok/s) is higher than the 32k cell and is listed on the build page as published. The RTX 4090 XQ file is an abliterated variant; the 57 tok/s row is the unmodified official-precision conversion on the same card. DGX Spark needle-row decode at long context comes from short answers; Tech2wild flags those figures as rough. The RTX 3090 rows use Tech2wild's own calibration-free INT4 checkpoint (Tech2wild/Inkling-2-Mini-W4A16) and a read-only overlay that ports the vLLM image's Inkling code to Ampere; the 2x 3090 headline uses DSpark, and the 157.4 tok/s eval-turn figure in that README is left out as tool-turn inflated.
Vendor-reported (not independently measured)
| Hardware | Decode tok/s | Prefill tok/s | Context | Quant | Engine | Kind | Source |
|---|---|---|---|---|---|---|---|
| 96 GB M5 Ultra Mac Studio | 210 | 3,620 | not stated | not stated | EXO upcoming Apple Silicon inference engine, with DSpark (follow-up post) | Vendor-reported | EXO Labs on X, 21 Nov 2026 |
| 48 GB M5 Max MacBook Pro | 125 | 1,750 | not stated | not stated | same EXO engine (follow-up post) | Vendor-reported | EXO Labs on X, 21 Nov 2026 |
These two rows are from EXO Labs' X post of 21 Nov 2026, crediting work with Thinking Machines. They are not in the measured build table and do not affect “fastest measured” rankings. A comment on llama.cpp issue #32707 also reported about 80 tok/s output and about 2,000 tok/s prefill on an RTX 5090 with Unsloth's UD-Q4_K_XL GGUF plus the DSpark draft file; that is an informal issue comment, not a formal bench, so it is not listed as a build.
Memory needed per quant
Thinking Machines' model card and Hugging Face page: one mixed NVFP4/MXFP8 checkpoint of about 20 GB; 32 GB or more of Apple silicon memory, or 32 GB or more of VRAM recommended; a single DGX Spark is a supported official setup. Unsloth's run-locally guide gives this starting range for RAM + VRAM, or unified memory (longer context, images and other apps use more; the mmproj-F16.gguf vision projector adds 639 MB):
| 1-bit | 2-bit | 3-bit | 4-bit | 8-bit (lossless) |
|---|---|---|---|---|
| 9 GB | 10–11 GB | 13–15 GB | 17–18 GB | 24 GB |
Recommended GGUF: UD-Q4_K_XL (17.6 GB). UD-Q8_K_XL (22.87 GB) is Unsloth's full-precision reference: official NVFP4 expert weights, the rest in BF16.
Unsloth also published a quality table for its GGUFs versus that UD-Q8_K_XL reference, with identical llama.cpp settings:
| Quant | Size | Mean KLD vs UD-Q8_K_XL | Top-1 % |
|---|---|---|---|
| UD-Q4_K_XL | 17.63 GB | 0.0107 | 98% |
| UD-Q4_K_M | 16.61 GB | 0.0108 | 97.9% |
| UD-IQ4_XS | 15.73 GB | 0.0109 | 97.7% |
| UD-Q3_K_XL | 13.63 GB | 0.0174 | 96.3% |
| UD-IQ3_XXS | 11.66 GB | 0.0269 | 94.7% |
| UD-Q2_K_XL | 9.14 GB | 0.0595 | 91.6% |
| UD-IQ2_XXS | 8.49 GB | 0.0726 | 90.6% |
| UD-IQ1_M | 8.02 GB | 0.0867 | 90.2% |
| UD-IQ1_S | 7.76 GB | 0.0911 | 89.4% |
Source: Unsloth, Inkling 2 Mini run-locally guide. Unsloth (quant publisher) run-locally guide, Nov 2026: each GGUF vs its UD-Q8_K_XL file (22.87 GB) across seven domains with identical llama.cpp settings. The reference keeps the official NVFP4 expert weights and stores the rest in BF16, i.e. the released precision, so it is the practical full-precision baseline for this model.
Measured process footprints, where the source reported them: pratikgx's IQ4_NL llama.cpp run on a 64 GB M5 Pro Mac mini peaked at 23.10 GB RSS at a 512-token prompt (reserved 128k KV slot) and 25.86 GB at 128k. The oMLX M5 Ultra oQ8e session peaked at 20.0 GB (1k context) to 59.4 GB (195k) on a 96 GB machine.
Which setup to pick
- 16 GB discrete GPU. A 4-bit or official file is tight (Unsloth's 4-bit range is 17–18 GB before context). The one 16 GB measurement in RigBase is an RX 9070 XT running InterpreticAI's IQ3_XXS GGUF (10.9 GiB) at 20.8 tok/s and 403 tok/s prefill in llama.cpp Vulkan, with no speculative decoding. On day-one kernels that was slower than dense Qwen3.8 27B IQ3_XXS on the same card (25.6 tok/s), which the poster blamed on an unfused relative-position bias. Expect this path to change as loaders improve.
- 24 GB (RTX 4090 class). This is the comfortable official-precision slot. IndrasMirror's Xolotl runs on an RTX 4090 measured 57 tok/s and 1,131 tok/s prefill for the exact 22.9 GB official conversion, and 59.7 tok/s / 1,162 prefill for a 13.93 GB XQ file, both at an 8K prompt with no DSpark in the figures. Official vLLM day-0 support targets Blackwell, Hopper and DGX Spark, not GeForce 4090 (vLLM blog, 20 Nov 2026).
- Used RTX 3090 (one or two cards). The official NVFP4/MXFP8 kernels need SM90+, so Ampere needs a different checkpoint. Tech2wild's recipe (MIT) re-quantizes the routed experts to INT4 (16.47 GB) and mounts a small overlay on the official vLLM image. On 2x RTX 3090 with an NVLink bridge (cards capped at 240 W) it measured 137 tok/s with DSpark at 458,752-token context, 100 tok/s with NVLink peer-to-peer off, and 89.8 tok/s with no speculative decoding. One RTX 3090 runs it at 73.5 tok/s (no spec decoding), but the 14.77 GiB of weights leave only 32K of context. Tech2wild reports prose perplexity about 2% above the original checkpoint, with code and math unchanged in its checks. See the multi-GPU 3090 guide.
- 32–64 GB Mac. The model card's laptop claim is realistic on memory: the official IQ4_NL GGUF (17.63 GB) ran on a 64 GB M5 Pro Mac mini at 29.95 tok/s short context and 17.39 tok/s at 128k, in llama.cpp, independently measured. That is usable chat speed, not the EXO peak.
- 96 GB M5 Ultra. Use the independent oMLX figures (83.4 tok/s at 4k, 56.2 at 32k, MTP on) when planning, and treat EXO's 210 tok/s as a vendor peak on an engine that was not public when we checked.
- DGX Spark (128 GB). The most complete long-context recipe in the set: Tech2wild's vLLM + official NVFP4/MXFP8 + DSpark build at 47.7 tok/s sustained on a 69-scenario eval (262K standard tier). Short-chat 72.8 tok/s was published as a peak and is not used as the headline. This is also the only sourced 1M-context run below.
Related hardware pages: M5 Ultra · M5 Max · M5 Pro · M3 Ultra · RTX 4090 · RTX 3090 · DGX Spark · Mac vs PC · MTP / DSpark speed-ups.
Long-context behaviour
The model card advertises up to 1M tokens. Only Tech2wild has published a sourced needle test that far. On one DGX Spark, official NVFP4/MXFP8, vLLM TP1, DSpark, 1M max FP8 KV tier:
- 199,989-token needle prompt: 47.2 tok/s decode, TTFT 118 s (1,693 tok/s prefill). Needle passed.
- 949,953-token needle prompt: 18.4 tok/s decode, TTFT 1,588 s (26.5 minutes), 598 tok/s average prefill. Needle passed (answered correctly at about 950K).
On 2x RTX 3090 (NVLink, INT4, DSpark), Tech2wild's needle at 439,384 tokens passed with a 190.9 s prefill (2,302 tok/s) and 119 tok/s decode after it; a 199,595-token needle took 59.3 s (3,367 tok/s). The KV cache is BF16 there, because FP8 KV for this model is SM12x-only.
Tech2wild's README says those needle-row decode figures come from short answers (77–149 tokens) and are rough. The 69-scenario eval headline at the 262K standard tier is still 47.7 tok/s, and the 1M max FP8 tier eval is 47.3 tok/s — so long-context prefill, not a collapse in decode, is the main wait. Credit: Tech2wild, Thinking Machines Lab, and the vLLM recipe.
On a much smaller machine, pratikgx's 64 GB M5 Pro Mac mini stayed in memory through a 131,072-token cell at 17.39 tok/s, but time to first token was 1,376 s (22.9 minutes). The oMLX M3 Ultra session (MTP off) declined smoothly from 40.3 tok/s at 1k to 28.4 tok/s at 195k.
How people are running it
From the model card and vendor docs, not from RigBase tests: on Apple silicon, Thinking Machines lists Ollama and Exo Labs; on NVIDIA, SGLang, vLLM, Unsloth, NVIDIA Dynamo and TokenSpeed; GGUF builds are available from Unsloth and llama.cpp (the official looped layout landed in llama.cpp PR #32694 / the ggml-org IQ4_NL file). Unsloth documents a separate looped-layer llama.cpp build for its GGUFs. vLLM ships a day-0 image, vllm/vllm-openai:inkling-2-mini, with native DSpark, officially for Blackwell, Hopper and DGX Spark. Day-one llama.cpp kernels are still catching up, which is why the RX 9070 XT IQ3_XXS figure is slower than a dense 27B on the same card.
Model page and calculator: Best home rig for Inkling 2 Mini · Can it run? · VRAM for Inkling 2 Mini. Launch post: Introducing Inkling 2 Mini.
FAQ
Can my rig run Inkling 2 Mini?
Often yes if you have about 24 GB of VRAM, or 32 GB or more of unified memory on Apple silicon. Thinking Machines' model card says the official checkpoint is about 20 GB (NVFP4/MXFP8) and recommends 32 GB or more on a Mac or a GPU. Unsloth's run-locally guide puts the recommended UD-Q4_K_XL GGUF at 17.6 GB and the lossless 8-bit file at 24 GB. A 16 GB card can run a smaller 3-bit GGUF: an RX 9070 XT measured 20.8 tok/s on an IQ3_XXS file (10.9 GiB) in day-one llama.cpp.
Is the 210 tok/s Mac Studio figure a measured RigBase result?
No. 210 tok/s decode and 3,620 tok/s prefill on a 96 GB M5 Ultra Mac Studio, and 125 / 1,750 tok/s on a 48 GB M5 Max MacBook Pro, are vendor-reported claims from EXO Labs, who said they worked with Thinking Machines. The only primary source we found is EXO's 21 Nov 2026 X post; it does not state the quant, context length or a public software version (EXO called it an upcoming Apple Silicon inference engine). An independent oMLX session on the same chip class (M5 Ultra, 96 GB) measured 83.4 tok/s at 4k context and 56.2 tok/s at 32k, with Lightning MTP on.
How much memory does Inkling 2 Mini need?
The official checkpoint is about 20 GB. Unsloth's hardware table (RAM + VRAM, or unified memory) starts at 9 GB for 1-bit, 10–11 GB for 2-bit, 13–15 GB for 3-bit, 17–18 GB for 4-bit and 24 GB for lossless 8-bit. Longer context, image input and the 639 MB vision projector add more. On a 64 GB M5 Pro Mac mini, pratikgx's llama.cpp IQ4_NL run used 23.10 GB process RSS at a 512-token prompt with a reserved 128k KV slot, and 25.86 GB at 128k.
How fast is it at long context, including 1M tokens?
On one DGX Spark, Tech2wild measured 47.2 tok/s decode after a 199,989-token needle prompt and 18.4 tok/s after a 949,953-token prompt on the 1M max FP8 tier. Time to first token at that 950K prompt was 1,588 s (26.5 minutes). The README says those needle-row decode figures come from short answers and are rough; the needle test passed. On a 64 GB M5 Pro Mac mini, pratikgx measured 17.39 tok/s at a 131,072-token prompt, with a 22.9 minute time to first token. On 2x RTX 3090 with NVLink (INT4, DSpark), Tech2wild passed a 439,384-token needle with a 190.9 s prefill and 119 tok/s decode after it.
Can an RTX 3090 run Inkling 2 Mini?
Yes, with a community INT4 checkpoint, because the official NVFP4/MXFP8 expert kernels need SM90+. Tech2wild's MIT-licensed recipe (Tech2wild/Inkling-2-Mini-W4A16 plus a small vLLM overlay for Ampere) measured 136.6 tok/s on 2x RTX 3090 with an NVLink bridge and DSpark (6-prompt bench, up to 458,752 tokens of context), 100.5 tok/s with NVLink peer-to-peer off, and 73.5 tok/s on one RTX 3090 with no speculative decoding (32K context at most).
Which quant should I use?
Unsloth recommends UD-Q4_K_XL (17.6 GB). Against its UD-Q8_K_XL reference (22.87 GB, official NVFP4 experts and BF16 for the rest), that file has mean KLD 0.0107 and 98.0% top-1. The official ~20 GB NVFP4/MXFP8 checkpoint is the native release format. Go below 4-bit only to fit a 16 GB card; Unsloth's UD-IQ3_XXS file is 11.66 GB (KLD 0.0269, 94.7% top-1).