Builds / RTX 3090
RTX 3090: Inkling 2 Mini W4A16 INT4 on vLLM TP1 + Ampere overlay, 73.5 tok/s median (Tech2wild)
Last updated:
- GPU(s)
- RTX 3090
Amazon · eBay (used) · Newegg - VRAM / unified memory (GB)
- 24
- CPU
- AMD Ryzen 9 5900X
Amazon · eBay (used) · Newegg - Motherboard
- not stated
- RAM
- 32 GB
Amazon · eBay (used) · Newegg - PSU
- not stated
- Other
- RTX 3090 (Ampere), one card of the same NVLink box
- Reported total cost
- not stated
- Power
- 240 W GPU power limit (stock 350 W)
Measured speed (generation tokens/sec)
| Model | Quant | Engine | tok/s | Prompt processing (tok/s) | Details |
|---|---|---|---|---|---|
| Inkling 2 Mini | W4A16 INT4 | vLLM TP1 + Ampere overlay, 1 stream | 73.5 | not stated | ⚡ spec-decode (off)context: 8,192 default, 32,768 max (33,392-token BF16 KV pool at gpu-memory-utilization 0.95)📏 Median decode with no speculative decoding, greedy, one request at a time, as published in the quality table of Tech2wild's README (the lane table says ~74 tok/s). Tech2wild's calibration-free INT4 re-quantization of the routed experts (Tech2wild/Inkling-2-Mini-W4A16, 16.47 GB + 0.64 GB DSpark drafter); the stock image needs Tech2wild's Ampere overlay. 14.77 GiB of weights on one 24 GB card, so context is short. 18/19 fixed prompts. |
Numbers by Tech2wild (tonyd2wild); Thinking Machines Lab (Inkling 2 Mini, DSpark drafter); vLLM project (vllm/vllm-openai:inkling-2-mini image, Humming expert kernels, Inkling 2 Mini recipe); compressed-tensors pack-quantized layout; Tech2wild/Inkling-2-Mini-W4A16 weights on Hugging Face. RigBase read the GitHub README on 2026-11-22 and checked each figure against the text. RigBase did not re-run it. Speculative decoding (MTP/DFlash/DSpark) inflates predictable text; these rows prefer prose/code over counting peaks. Recipe license: MIT.
https://github.com/tonyd2wild/Inkling-2-Mini-W4A16-RTX3090/blob/main/README.md
https://huggingface.co/Tech2wild/Inkling-2-Mini-W4A16
Blank fields = not stated in the source. We never estimate build data; GPU street prices above are separately sourced and dated.
Quick answers: Can RTX 3090 run Inkling 2 Mini?
Speculative-decoding rows on this page are not like-for-like with plain decode. See how much MTP and DFlash really speed things up (measured pairs).