More
Submit build

Guides / Inkling 2 Mini

Can my rig run Inkling 2 Mini?

Quick answer

Can my rig run Inkling 2 Mini?
Often yes if you have about 24 GB of VRAM, or 32 GB or more of unified memory on Apple silicon. Thinking Machines' model card says the official checkpoint is about 20 GB (NVFP4/MXFP8) and recommends 32 GB or more on a Mac or a GPU. Unsloth's run-locally guide puts the recommended UD-Q4_K_XL GGUF at 17.6 GB and the lossless 8-bit file at 24 GB. A 16 GB card can run a smaller 3-bit GGUF: an RX 9070 XT measured 20.8 tok/s on an IQ3_XXS file (10.9 GiB) in day-one llama.cpp.

Last updated:

Inkling 2 Mini is Thinking Machines Lab's 20 November 2026 open-weights model: a looped mixture-of-experts transformer with 26.4B total parameters, 3.4B active, and about 5.4B effective active parameters per token because the middle 12 layers run three times. The official checkpoint is already NVFP4/MXFP8, about 20 GB, with up to 1M context, and it is released under Apache 2.0. This page collects every sourced local speed RigBase has, and separates vendor peak claims from independent measurements.

Quick answer: Plan on about 24 GB of VRAM or 32 GB or more of unified memory for a comfortable 4-bit or official-precision run. Independently measured decode today ranges from 20.8 tok/s on an RX 9070 XT 16 GB (day-one llama.cpp, IQ3_XXS) to 83.4 tok/s on a 96 GB M5 Ultra in oMLX with Lightning MTP, and 59.7 tok/s on an RTX 4090 with the Xolotl engine (no drafter in that figure). On two used RTX 3090s with an NVLink bridge, Tech2wild's INT4 vLLM recipe with DSpark measured 137 tok/s (6-prompt bench) at up to 458,752 tokens of context. EXO Labs reported 210 tok/s on a 96 GB M5 Ultra Mac Studio and 125 tok/s on a 48 GB M5 Max MacBook Pro; those are vendor claims from an X post, not independently measured, and the post does not state the quant or context length. Full results: Inkling 2 Mini model page.

Hype vs measured

The model launched with a lot of attention. Thinking Machines' announcement (20 November 2026) and X post describe a looped transformer that is “small enough to run on your laptop,” scores 20.1 on the AA Index, and is much smaller than Inkling 2 Small (594B parameters). The lab also said researchers ran it offline on a MacBook Pro. Those statements are from the lab, not from RigBase measurements.

The same day, EXO Labs posted that they had worked with Thinking Machines to make Inkling 2 Mini run fast locally:

A follow-up on the same thread said EXO got access on Wednesday, had it running at 30 tok/s in three hours, and at 210 tok/s a day later on an upcoming Apple Silicon inference engine, with architecture-specific kernel work and “better use of DSpark with adaptive drafting.” RigBase looked for a blog post, GitHub README or model card behind those figures on 21 Nov 2026 (blog.exolabs.net and benchmarks.exolabs.net had no Inkling 2 Mini write-up). The X posts do not state the quant, the context length, or a public software version. Treat 210 and 125 as vendor-reported peaks from EXO, with Thinking Machines, not independently measured.

Independent numbers on the same chip class are lower. A public oMLX session on a 96 GB M5 Ultra (oQ8e, Lightning MTP on) measured 83.4 tok/s at 4k context and 56.2 tok/s at 32k, with prefill of 3,225 and 2,968 tok/s. On a 64 GB M5 Pro Mac mini, pratikgx measured 29.95 tok/s in llama.cpp with the official IQ4_NL GGUF at a 512-token prompt, with no speculative decoding. Looping is the main reason a “3.4B-active” model is not as fast as a typical small MoE: each token runs the middle layers three times (~5.4B effective active parameters), as the model card explains.

Sourced speed by hardware

Generation (decode) tok/s is not comparable 1:1 across quants, engines or speculative decoding. Prefill is prompt-processing speed at the context the source measured. 31 measured results from 8 builds are in RigBase today; the table below shows the headline and long-context rows. Every number links to its build page and original source.

HardwareDecode tok/sPrefill tok/sContextQuantEngineKindSource
Mac Studio M5 Ultra 96 GB (80-core)83.43,225 @4k4k tokensoMLX oQ8e, Affine8 8-bit KVoMLX v0.8.1, Lightning MTPIndependentsource
Mac Studio M5 Ultra 96 GB (80-core)56.22,968 @32k32k tokensoMLX oQ8e, Affine8 8-bit KVoMLX v0.8.1, Lightning MTPIndependentsource
RTX 4090 24 GB59.71,162 @8K8K prompt, 1,024 generatedXQ mixed (~4.0 bpw routed experts, 13 of 24 layers keep official NVFP4; Q8_0 bacXolotl (xolotl-inkling)Independentsource
RTX 4090 24 GB571,131 @8K8K prompt, 1,024 generatedOfficial precision, exact conversion (NVFP4 + MXFP8 experts, BF16 rest; 22.9 GB)Xolotl (xolotl-inkling)Independentsource
2x RTX 3090 24 GB, NVLink137–458,752 per request (TP2 max lane, text-only): BF16 KV, 483,850-token poolW4A16 INT4 + DSparkvLLM TP2 + Ampere overlay, NVLink, 1 streamIndependentsource
2x RTX 3090 24 GB, NVLink1192,302 @439,384439,384-token needle prompt on the TP2 max lane (458,752 per request)W4A16 INT4 + DSparkvLLM TP2 + Ampere overlay, NVLink, 1 streamIndependentsource
2x RTX 3090 24 GB (no spec decoding)89.8–not stated (TP2 quality comparison, greedy)W4A16 INT4vLLM TP2 + Ampere overlay, 1 streamIndependentsource
RTX 3090 24 GB (one card)73.5–8,192 default, 32,768 max (33,392-token BF16 KV pool at gpu-memory-utilization 0.95)W4A16 INT4vLLM TP1 + Ampere overlay, 1 streamIndependentsource
DGX Spark (128 GB unified)47.7–262K standard tier: BF16 KV, gpu-memory-utilization 0.80 (68.89 GiB KV, 1,346,671-token pool)NVFP4/MXFP8 + DSparkvLLM TP1, 1 streamIndependentsource
DGX Spark (128 GB unified)47.21,693 @199,989199,989-token prompt on the 1M max FP8 tier (74 GiB FP8 KV)NVFP4/MXFP8 + DSpark, FP8 KVvLLM TP1, 1 streamIndependentsource
DGX Spark (128 GB unified)18.4598 @949,953949,953-token prompt on the 1M max FP8 tier (74 GiB FP8 KV)NVFP4/MXFP8 + DSpark, FP8 KVvLLM TP1, 1 streamIndependentsource
Mac Studio M3 Ultra 512 GB (80-core)40.31,169 @1k1k tokensoMLX Inkling-2-Mini (as labelled)oMLX v0.8.1, MTP offIndependentsource
Mac mini M5 Pro 64 GB29.95491 @512512-token prompt, 256 generatedggml-org IQ4_NL (17.63 GB; 4-bit experts in layers 0-19, Q8_0 elsewhere, loop collama.cpp PR #32694 (d7cb78c91, build 12587), MetalIndependentsource
Mac mini M5 Pro 64 GB17.3995.29 @131,072131,072-token prompt, 256 generatedggml-org IQ4_NL (17.63 GB; 4-bit experts in layers 0-19, Q8_0 elsewhere, loop collama.cpp PR #32694 (d7cb78c91, build 12587), MetalIndependentsource
RX 9070 XT 16 GB (Windows)20.8403 @2048llama-bench tg64IQ3_XXS (InterpreticAI, 10.9 GiB)llama.cpp b12600 Vulkan (Windows)Independentsource

oMLX M5 Ultra rows use Lightning MTP; the 128k generation figure in that session (74.7 tok/s) is higher than the 32k cell and is listed on the build page as published. The RTX 4090 XQ file is an abliterated variant; the 57 tok/s row is the unmodified official-precision conversion on the same card. DGX Spark needle-row decode at long context comes from short answers; Tech2wild flags those figures as rough. The RTX 3090 rows use Tech2wild's own calibration-free INT4 checkpoint (Tech2wild/Inkling-2-Mini-W4A16) and a read-only overlay that ports the vLLM image's Inkling code to Ampere; the 2x 3090 headline uses DSpark, and the 157.4 tok/s eval-turn figure in that README is left out as tool-turn inflated.

Vendor-reported (not independently measured)

HardwareDecode tok/sPrefill tok/sContextQuantEngineKindSource
96 GB M5 Ultra Mac Studio2103,620not statednot statedEXO upcoming Apple Silicon inference engine, with DSpark (follow-up post)Vendor-reportedEXO Labs on X, 21 Nov 2026
48 GB M5 Max MacBook Pro1251,750not statednot statedsame EXO engine (follow-up post)Vendor-reportedEXO Labs on X, 21 Nov 2026

These two rows are from EXO Labs' X post of 21 Nov 2026, crediting work with Thinking Machines. They are not in the measured build table and do not affect “fastest measured” rankings. A comment on llama.cpp issue #32707 also reported about 80 tok/s output and about 2,000 tok/s prefill on an RTX 5090 with Unsloth's UD-Q4_K_XL GGUF plus the DSpark draft file; that is an informal issue comment, not a formal bench, so it is not listed as a build.

Memory needed per quant

Thinking Machines' model card and Hugging Face page: one mixed NVFP4/MXFP8 checkpoint of about 20 GB; 32 GB or more of Apple silicon memory, or 32 GB or more of VRAM recommended; a single DGX Spark is a supported official setup. Unsloth's run-locally guide gives this starting range for RAM + VRAM, or unified memory (longer context, images and other apps use more; the mmproj-F16.gguf vision projector adds 639 MB):

1-bit2-bit3-bit4-bit8-bit (lossless)
9 GB10–11 GB13–15 GB17–18 GB24 GB

Recommended GGUF: UD-Q4_K_XL (17.6 GB). UD-Q8_K_XL (22.87 GB) is Unsloth's full-precision reference: official NVFP4 expert weights, the rest in BF16.

Unsloth also published a quality table for its GGUFs versus that UD-Q8_K_XL reference, with identical llama.cpp settings:

QuantSizeMean KLD vs UD-Q8_K_XLTop-1 %
UD-Q4_K_XL17.63 GB0.010798%
UD-Q4_K_M16.61 GB0.010897.9%
UD-IQ4_XS15.73 GB0.010997.7%
UD-Q3_K_XL13.63 GB0.017496.3%
UD-IQ3_XXS11.66 GB0.026994.7%
UD-Q2_K_XL9.14 GB0.059591.6%
UD-IQ2_XXS8.49 GB0.072690.6%
UD-IQ1_M8.02 GB0.086790.2%
UD-IQ1_S7.76 GB0.091189.4%

Source: Unsloth, Inkling 2 Mini run-locally guide. Unsloth (quant publisher) run-locally guide, Nov 2026: each GGUF vs its UD-Q8_K_XL file (22.87 GB) across seven domains with identical llama.cpp settings. The reference keeps the official NVFP4 expert weights and stores the rest in BF16, i.e. the released precision, so it is the practical full-precision baseline for this model.

Measured process footprints, where the source reported them: pratikgx's IQ4_NL llama.cpp run on a 64 GB M5 Pro Mac mini peaked at 23.10 GB RSS at a 512-token prompt (reserved 128k KV slot) and 25.86 GB at 128k. The oMLX M5 Ultra oQ8e session peaked at 20.0 GB (1k context) to 59.4 GB (195k) on a 96 GB machine.

Which setup to pick

Related hardware pages: M5 Ultra · M5 Max · M5 Pro · M3 Ultra · RTX 4090 · RTX 3090 · DGX Spark · Mac vs PC · MTP / DSpark speed-ups.

Long-context behaviour

The model card advertises up to 1M tokens. Only Tech2wild has published a sourced needle test that far. On one DGX Spark, official NVFP4/MXFP8, vLLM TP1, DSpark, 1M max FP8 KV tier:

On 2x RTX 3090 (NVLink, INT4, DSpark), Tech2wild's needle at 439,384 tokens passed with a 190.9 s prefill (2,302 tok/s) and 119 tok/s decode after it; a 199,595-token needle took 59.3 s (3,367 tok/s). The KV cache is BF16 there, because FP8 KV for this model is SM12x-only.

Tech2wild's README says those needle-row decode figures come from short answers (77–149 tokens) and are rough. The 69-scenario eval headline at the 262K standard tier is still 47.7 tok/s, and the 1M max FP8 tier eval is 47.3 tok/s — so long-context prefill, not a collapse in decode, is the main wait. Credit: Tech2wild, Thinking Machines Lab, and the vLLM recipe.

On a much smaller machine, pratikgx's 64 GB M5 Pro Mac mini stayed in memory through a 131,072-token cell at 17.39 tok/s, but time to first token was 1,376 s (22.9 minutes). The oMLX M3 Ultra session (MTP off) declined smoothly from 40.3 tok/s at 1k to 28.4 tok/s at 195k.

How people are running it

From the model card and vendor docs, not from RigBase tests: on Apple silicon, Thinking Machines lists Ollama and Exo Labs; on NVIDIA, SGLang, vLLM, Unsloth, NVIDIA Dynamo and TokenSpeed; GGUF builds are available from Unsloth and llama.cpp (the official looped layout landed in llama.cpp PR #32694 / the ggml-org IQ4_NL file). Unsloth documents a separate looped-layer llama.cpp build for its GGUFs. vLLM ships a day-0 image, vllm/vllm-openai:inkling-2-mini, with native DSpark, officially for Blackwell, Hopper and DGX Spark. Day-one llama.cpp kernels are still catching up, which is why the RX 9070 XT IQ3_XXS figure is slower than a dense 27B on the same card.

Model page and calculator: Best home rig for Inkling 2 Mini · Can it run? · VRAM for Inkling 2 Mini. Launch post: Introducing Inkling 2 Mini.

FAQ

Can my rig run Inkling 2 Mini?

Often yes if you have about 24 GB of VRAM, or 32 GB or more of unified memory on Apple silicon. Thinking Machines' model card says the official checkpoint is about 20 GB (NVFP4/MXFP8) and recommends 32 GB or more on a Mac or a GPU. Unsloth's run-locally guide puts the recommended UD-Q4_K_XL GGUF at 17.6 GB and the lossless 8-bit file at 24 GB. A 16 GB card can run a smaller 3-bit GGUF: an RX 9070 XT measured 20.8 tok/s on an IQ3_XXS file (10.9 GiB) in day-one llama.cpp.

Is the 210 tok/s Mac Studio figure a measured RigBase result?

No. 210 tok/s decode and 3,620 tok/s prefill on a 96 GB M5 Ultra Mac Studio, and 125 / 1,750 tok/s on a 48 GB M5 Max MacBook Pro, are vendor-reported claims from EXO Labs, who said they worked with Thinking Machines. The only primary source we found is EXO's 21 Nov 2026 X post; it does not state the quant, context length or a public software version (EXO called it an upcoming Apple Silicon inference engine). An independent oMLX session on the same chip class (M5 Ultra, 96 GB) measured 83.4 tok/s at 4k context and 56.2 tok/s at 32k, with Lightning MTP on.

How much memory does Inkling 2 Mini need?

The official checkpoint is about 20 GB. Unsloth's hardware table (RAM + VRAM, or unified memory) starts at 9 GB for 1-bit, 10–11 GB for 2-bit, 13–15 GB for 3-bit, 17–18 GB for 4-bit and 24 GB for lossless 8-bit. Longer context, image input and the 639 MB vision projector add more. On a 64 GB M5 Pro Mac mini, pratikgx's llama.cpp IQ4_NL run used 23.10 GB process RSS at a 512-token prompt with a reserved 128k KV slot, and 25.86 GB at 128k.

How fast is it at long context, including 1M tokens?

On one DGX Spark, Tech2wild measured 47.2 tok/s decode after a 199,989-token needle prompt and 18.4 tok/s after a 949,953-token prompt on the 1M max FP8 tier. Time to first token at that 950K prompt was 1,588 s (26.5 minutes). The README says those needle-row decode figures come from short answers and are rough; the needle test passed. On a 64 GB M5 Pro Mac mini, pratikgx measured 17.39 tok/s at a 131,072-token prompt, with a 22.9 minute time to first token. On 2x RTX 3090 with NVLink (INT4, DSpark), Tech2wild passed a 439,384-token needle with a 190.9 s prefill and 119 tok/s decode after it.

Can an RTX 3090 run Inkling 2 Mini?

Yes, with a community INT4 checkpoint, because the official NVFP4/MXFP8 expert kernels need SM90+. Tech2wild's MIT-licensed recipe (Tech2wild/Inkling-2-Mini-W4A16 plus a small vLLM overlay for Ampere) measured 136.6 tok/s on 2x RTX 3090 with an NVLink bridge and DSpark (6-prompt bench, up to 458,752 tokens of context), 100.5 tok/s with NVLink peer-to-peer off, and 73.5 tok/s on one RTX 3090 with no speculative decoding (32K context at most).

Which quant should I use?

Unsloth recommends UD-Q4_K_XL (17.6 GB). Against its UD-Q8_K_XL reference (22.87 GB, official NVFP4 experts and BF16 for the rest), that file has mean KLD 0.0107 and 98.0% top-1. The official ~20 GB NVFP4/MXFP8 checkpoint is the native release format. Go below 4-bit only to fit a 16 GB card; Unsloth's UD-IQ3_XXS file is 11.66 GB (KLD 0.0269, 94.7% top-1).