Builds / 2x RTX 3090
2x RTX 3090: Inkling 2 Mini W4A16 INT4 on vLLM TP2 + Ampere overlay, 136.6 tok/s bench (Tech2wild)
Last updated:
- GPU(s)
- 2x RTX 3090
Amazon · eBay (used) · Newegg - VRAM / unified memory (GB)
- 48
- CPU
- AMD Ryzen 9 5900X
Amazon · eBay (used) · Newegg - Motherboard
- not stated
- RAM
- 32 GB
Amazon · eBay (used) · Newegg - PSU
- not stated
- Other
- 2× RTX 3090 (Ampere) joined by an NVLink bridge (NV2); both cards on x4 PCIe links
- Reported total cost
- not stated
- Power
- 240 W GPU power limit per card (stock 350 W)
Measured speed (generation tokens/sec)
| Model | Quant | Engine | tok/s | Prompt processing (tok/s) | Details |
|---|---|---|---|---|---|
| Inkling 2 Mini | W4A16 INT4 + DSpark | vLLM TP2 + Ampere overlay, NVLink, 1 stream | 136.6 | not stated | ⚡ spec-decode (DSpark (num_speculative_tokens 7))context: 458,752 per request (TP2 max lane, text-only): BF16 KV, 483,850-token pool📏 bench/bench_decode.py: 6 fixed prompts, temperature 0, reasoning effort none, streaming, one request at a time; 136.6 tok/s aggregate (median 129.7) as published in Tech2wild's README. Tech2wild's calibration-free INT4 re-quantization of the routed experts (Tech2wild/Inkling-2-Mini-W4A16, 16.47 GB + 0.64 GB DSpark drafter); the stock image needs Tech2wild's Ampere overlay. The 137.5 tok/s table headline is the same lane; the 157.4 tok/s eval-turn decode (tool turns, 3.31 tokens per DSpark step) is not used. NVLink bridge on, cards capped at 240 W. 69-scenario tool eval quality 87.0/100. |
| Inkling 2 Mini | W4A16 INT4 + DSpark | vLLM TP2 + Ampere overlay, NVLink P2P off (NCCL_P2P_DISABLE=1), 1 stream | 100.5 | not stated | ⚡ spec-decode (DSpark (num_speculative_tokens 7))context: 458,752 per request (TP2 max lane)📏 Single-stream decode on the same lane with NCCL peer-to-peer disabled, as published in Tech2wild's README (137.5 with NVLink, +37%). Both cards report x4 PCIe links, so the README says a board with x16 slots may do better without a bridge. |
| Inkling 2 Mini | W4A16 INT4 | vLLM TP2 + Ampere overlay, 1 stream | 89.8 | not stated | ⚡ spec-decode (off)context: not stated (TP2 quality comparison, greedy)📏 Median decode with no speculative decoding, greedy, as published in the quality table of Tech2wild's README (original NVFP4/MXFP8 checkpoint on the same cards and overlay: 87.2 tok/s). The lane table lists ~90 tok/s for the 131K vision lane, DSpark off. |
| Inkling 2 Mini | W4A16 INT4 + DSpark | vLLM TP2 + Ampere overlay, NVLink, 1 stream | 119 | 2,302 @ 439,384-token prompt est. TTFT @8k ≈ 3.6 s (calculated from measured prefill) measured TTFT: 191 s @ 439,384-token prompt | ⚡ spec-decode (DSpark (num_speculative_tokens 7))context: 439,384-token needle prompt on the TP2 max lane (458,752 per request)📏 Single-request needle test (passphrase at 50% depth in synthetic log text) as published in Tech2wild's README; decode after the prompt. Needle PASS. Other needles: ~32K at 3,068 tok/s prefill and 199,595 tokens in 59.3 s (3,367 tok/s), both PASS. |
Prompt processing = prefill speed (how fast the prompt is read) as reported by the source, with the prompt length it was measured at; blank = the source does not report it. est. TTFT @8k is calculated from the measured prefill speed (8,192 ÷ prefill tok/s), shown only when prefill was measured on a prompt of 512+ tokens. It is not a measurement and ignores load time and caching. measured TTFT is the source's own time-to-first-token. Why prefill matters for coding assistants.
Numbers by Tech2wild (tonyd2wild); Thinking Machines Lab (Inkling 2 Mini, DSpark drafter); vLLM project (vllm/vllm-openai:inkling-2-mini image, Humming expert kernels, Inkling 2 Mini recipe); compressed-tensors pack-quantized layout; Tech2wild/Inkling-2-Mini-W4A16 weights on Hugging Face. RigBase read the GitHub README on 2026-11-22 and checked each figure against the text. RigBase did not re-run it. Speculative decoding (MTP/DFlash/DSpark) inflates predictable text; these rows prefer prose/code over counting peaks. Recipe license: MIT.
https://github.com/tonyd2wild/Inkling-2-Mini-W4A16-RTX3090/blob/main/README.md
https://huggingface.co/Tech2wild/Inkling-2-Mini-W4A16
Blank fields = not stated in the source. We never estimate build data; GPU street prices above are separately sourced and dated.
Quick answers: Can 2x RTX 3090 run Inkling 2 Mini?
Speculative-decoding rows on this page are not like-for-like with plain decode. See how much MTP and DFlash really speed things up (measured pairs).