Best local LLMs for 128 GB or more of VRAM or unified memory
Measured speeds from builds where the model ran on more than 96 GB of GPU or unified memory. Each number links to its build and source.
8 GB · 12 GB · 16 GB · 24 GB · 32 GB · 48 GB · 64 GB · 96 GB · 128 GB or more
GPUs in this tier: DGX Spark / GB10
Fastest measured result per model
| Model | Plain decode tok/s | With speculative decoding ⚡ | Results |
|---|---|---|---|
| Qwen3.6 35B-A3B | 107 (4-bit MLX, Apple M5 Max (unified)) | 271 (4-bit MLX (mlx-serve build), Apple M5 Max (unified)) | 17 |
| Qwen3.8 27B | 31 (4-bit MLX, Apple M5 Max (unified)) | 45.3 (4-bit MLX + Qwen3.8-27B-MTP-4bit drafter, Apple M5 Max (40-core GPU, unified)) | 16 |
| gpt-oss 20B | 116.08 (MXFP4, Apple M2 Ultra) | – | 6 |
| gpt-oss 120B | 79.68 (MXFP4, Apple M2 Ultra) | – | 5 |
| Gemma 4 26B-A4B | 51.8 (4-bit, Strix Halo (Radeon 8060S)) | 166 (4-bit MLX (QAT), Apple M5 Max (unified)) | 4 |
| GLM-5.3 Flash | 105.3 (NVFP4, 4x DGX Spark (GB10)) | – | 4 |
| Muse Spark Open 816B-A42B | 21 (oQ3e (3-bit), ~344 GB, Apple M3 Ultra) | – | 2 |
| Mistral Medium 3.5 128B | 7.7 (FP8, 8,192-token context, Apple M5 Ultra) | – | 2 |
| Nemotron 3.5 Super VL 120B-A12B | 19.06 (NVFP4, no speculative decoding, 256-token output, DGX Spark (GB10)) | 38.07 (NVFP4 + LiLiCorr speculative decoding (7 draft tokens), DGX Spark (GB10)) | 2 |
| Nemotron 3.5 Bolt 74B-A9B | 39.47 (2-bit 24GB GGUF, no speculative decoding, DGX Spark (GB10)) | 61.61 (2-bit 24GB GGUF + DFlash speculative decoding (7 draft tokens), DGX Spark (GB10)) | 2 |
| Nemotron 3.5 Lightning 30B-A3B | 70.3 (no speculative decoding, DGX Spark (GB10)) | 94.4 (+ DSpark speculative decoding (k=3), DGX Spark (GB10)) | 2 |
| Nemotron 3 Nano Omni 30B | 58.8 (, DGX Spark (GB10)) | – | 1 |
| Qwen3.8 Flash-Next | – | 26.8 (+ MTP (k=2), DGX Spark (GB10)) | 1 |
| Gemma 4 31B | 8.8 (4-bit, Strix Halo (Radeon 8060S)) | – | 1 |
| DeepSeek V4.1 Flash | – | 30.8 (not stated, Apple M3 Ultra) | 1 |
| MiniMax M3.1 Flash | 25.3 (oQ3e-mtp (MTP off), Apple M3 Ultra) | – | 1 |
| Kolibri-1 | 58.6 (Q4_K_M (47.45 GB), Strix Halo (Radeon 8060S)) | – | 1 |
| Step 5 Preview | – | 25 (EXL3 (experts 4.01 bpw, 341.4 GB total), 4x DGX Spark (GB10)) | 1 |
| DeepSeek V4 Flash legacy | 95.52 (see source (FP8 KV cache), 8x RTX 3090) | – | 3 |
| Qwen3.6 27B legacy | 11.4 (4-bit, Strix Halo (Radeon 8060S)) | 32.83 (NVFP4, DGX Spark (GB10)) | 3 |
| DeepSeek R1 671B legacy | 19.89 (, Apple M3 Ultra) | – | 2 |
| Llama 3.1 70B legacy | 14.08 (, Apple M3 Ultra) | – | 2 |
| DeepSeek R1 Distill 14B legacy | 51.85 (, Apple M3 Ultra) | – | 2 |
| Llama 3.2 3B legacy | 154.6 (, Apple M3 Ultra) | – | 2 |
| Llama 3.1 8B legacy | 56.5 (MLPerf Client v2.0 (geomean), RTX Spark N1X) | – | 2 |
| Nemotron 3 Super 120B-A12B legacy | 16.1 (no speculative decoding, DGX Spark (GB10)) | 23.0 (+ MTP (k=3), DGX Spark (GB10)) | 2 |
| MiniMax M2.7 legacy | 29.5 (IQ4_XS, AMD Ryzen AI Max+ PRO 495 (Radeon 8065S iGPU, 40 CUs)) | – | 2 |
| Llama 2 7B legacy | 125.21 (Q4_0, Apple M2 Ultra) | – | 1 |
| Llama 3.1 405B legacy | 1.96 (, Apple M3 Ultra) | – | 1 |
| GLM-5.2 legacy | – | 32.5 (QuantTrio Int4-Int8Mix, 4x DGX Spark (GB10)) | 1 |
| MiniMax M3 legacy | 28.9 (4-bit MLX, Apple M3 Ultra) | – | 1 |
Excludes results the source marks pending and results where part of the model sat in system RAM.
All results in this tier
(MTP, DFlash, DSpark and similar draft-and-verify methods give much higher tok/s on predictable text and are not comparable 1:1 with plain decode)
| Model | Build | Ran on | Quant | Engine | tok/s | Details |
|---|---|---|---|---|---|---|
| DeepSeek V4.1 Flash | Mac Studio M3 Ultra 512 GB running DeepSeek V4.1 Flash with MTP | Apple M3 Ultra (512 GB) | not stated | oMLX v0.7.1.dev1, MTP enabled | 30.8 | ⚡ spec-decode (MTP)prefill 541.7 tok/scontext: 1K (36.0 tok/s at 8K)📏 oMLX community benchmark upload, peak memory 289.3 GB |
| GLM-5.3 Flash | 4x DGX Spark ring running GLM-5.3 Flash NVFP4 | 4x DGX Spark (GB10) (512 GB) | NVFP4 | vLLM TP4 (RiNGSiDE), single-stream code | 105.3 | 📏 Single-stream decode reported by the source |
| GLM-5.3 Flash | 4x DGX Spark ring running GLM-5.3 Flash NVFP4 | 4x DGX Spark (GB10) (512 GB) | NVFP4 | vLLM TP4 (RiNGSiDE), single-stream prose | 61.7 | 📏 Single-stream decode reported by the source |
| GLM-5.3 Flash | 2x DGX Spark running GLM-5.3 Flash NVFP4 | 2x DGX Spark (GB10) (256 GB) | NVFP4 | vLLM TP2 (RiNGSiDE), single-stream code | 56.5 | 📏 Single-stream decode reported by the source |
| GLM-5.3 Flash | 2x DGX Spark running GLM-5.3 Flash NVFP4 | 2x DGX Spark (GB10) (256 GB) | NVFP4 | vLLM TP2 (RiNGSiDE), single-stream prose | 33.0 | 📏 Single-stream decode reported by the source |
| Gemma 4 26B-A4B | MacBook Pro M5 Max 128 GB: Qwen3.6 35B-A3B at 271 tok/s (mlx-serve + MTP), Gemma 4 26B-A4B at 166 | Apple M5 Max (unified) (128 GB) | 4-bit MLX (QAT) | Rapid-MLX 0.16.1 + DFlash | 166 | ⚡ spec-decode (DFlash) |
| Gemma 4 26B-A4B | DGX Spark (GB10, 128 GB): 23 models on vLLM at 128K context, Qwen3.6 35B-A3B up to 102 tok/s | DGX Spark (GB10) (128 GB) | + official MTP drafter (k=3) | vLLM 0.31.0, 128K ctx, 1 user | 56.5 | ⚡ spec-decode (MTP) |
| Gemma 4 26B-A4B | HP ZBook Ultra G1a (Strix Halo laptop, 128 GB): Qwen3.6 35B-A3B at 59.4 tok/s, gpt-oss 120B at 44.7 | Strix Halo (Radeon 8060S) (128 GB) | 4-bit | Ollama 0.34.0 | 51.8 | |
| Gemma 4 26B-A4B | DGX Spark (GB10, 128 GB): 23 models on vLLM at 128K context, Qwen3.6 35B-A3B up to 102 tok/s | DGX Spark (GB10) (128 GB) | no speculative decoding | vLLM 0.31.0, 128K ctx, 1 user | 30.8 | |
| Gemma 4 31B | HP ZBook Ultra G1a (Strix Halo laptop, 128 GB): Qwen3.6 35B-A3B at 59.4 tok/s, gpt-oss 120B at 44.7 | Strix Halo (Radeon 8060S) (128 GB) | 4-bit | Ollama 0.34.0 | 8.8 | |
| Kolibri-1 | Strix Halo (Ryzen AI Max+ 395, 128 GB) running Kolibri-1 Q4_K_M | Strix Halo (Radeon 8060S) (128 GB) | Q4_K_M (47.45 GB) | llama.cpp ROCm (Seraphiel102 fork 8798b65), -ngl 99, f16 KV | 58.6 | prefill 951 tok/scontext: 512 (30.5 tok/s at 131K; prefill measured at 32K)📏 C3MA AI_Benchmarks repository, commit 22c9f2dcf2 (2026-10-19) |
| MiniMax M3.1 Flash | Mac Studio M3 Ultra (60-core GPU) 256 GB running MiniMax M3.1 Flash | Apple M3 Ultra (256 GB) | oQ3e-mtp (MTP off) | oMLX v0.7.2 | 25.3 | prefill 290.1 tok/scontext: 1K (18.0 tok/s at 4K)📏 oMLX community benchmark upload, mtp_enabled false, peak memory 182.1 GB |
| Mistral Medium 3.5 128B | Mac Studio M5 Ultra 512GB: Qwen3.8 27B and Mistral Medium 3.5 (Apple MLX benchmark) | Apple M5 Ultra (512 GB) | FP8, 8,192-token context | MLX (mlx_lm.benchmark) | 7.7 | |
| Mistral Medium 3.5 128B | Mac Studio M3 Ultra 512GB: Qwen3.8 27B and Mistral Medium 3.5 (Apple MLX benchmark) | Apple M3 Ultra (512 GB) | FP8, 8,192-token context | MLX (mlx_lm.benchmark) | 4.95 | |
| Muse Spark Open 816B-A42B | Mac Studio M3 Ultra 512GB: Muse Spark Open 816B (Hugging Face test) | Apple M3 Ultra (512 GB) | oQ3e (3-bit), ~344 GB | MLX (oMLX / mlx-lm) | 21 | |
| Muse Spark Open 816B-A42B | Mac Studio M3 Ultra 512GB: Muse Spark Open 816B (Hugging Face test) | Apple M3 Ultra (512 GB) | Q2_K (imatrix), ~304 GB | llama.cpp | 18 | |
| Nemotron 3 Nano Omni 30B | DGX Spark (GB10, 128 GB): 23 models on vLLM at 128K context, Qwen3.6 35B-A3B up to 102 tok/s | DGX Spark (GB10) (128 GB) | vLLM 0.31.0, 128K ctx, 1 user | 58.8 | ||
| Nemotron 3.5 Bolt 74B-A9B | DGX Spark (GB10, 128GB): Nemotron 3.5 Super VL 120B and Bolt 74B, launch-week runs | DGX Spark (GB10) (128 GB) | 2-bit 24GB GGUF + DFlash speculative decoding (7 draft tokens) | llama.cpp | 61.61 | ⚡ spec-decode (DFlash) |
| Nemotron 3.5 Bolt 74B-A9B | DGX Spark (GB10, 128GB): Nemotron 3.5 Super VL 120B and Bolt 74B, launch-week runs | DGX Spark (GB10) (128 GB) | 2-bit 24GB GGUF, no speculative decoding | llama.cpp | 39.47 | |
| Nemotron 3.5 Lightning 30B-A3B | DGX Spark (GB10, 128 GB): 23 models on vLLM at 128K context, Qwen3.6 35B-A3B up to 102 tok/s | DGX Spark (GB10) (128 GB) | + DSpark speculative decoding (k=3) | vLLM 0.31.0, 128K ctx, 1 user | 94.4 | ⚡ spec-decode (DSpark) |
| Nemotron 3.5 Lightning 30B-A3B | DGX Spark (GB10, 128 GB): 23 models on vLLM at 128K context, Qwen3.6 35B-A3B up to 102 tok/s | DGX Spark (GB10) (128 GB) | no speculative decoding | vLLM 0.31.0, 128K ctx, 1 user | 70.3 | |
| Nemotron 3.5 Super VL 120B-A12B | DGX Spark (GB10, 128GB): Nemotron 3.5 Super VL 120B and Bolt 74B, launch-week runs | DGX Spark (GB10) (128 GB) | NVFP4 + LiLiCorr speculative decoding (7 draft tokens) | vLLM | 38.07 | ⚡ spec-decode (LiLiCorr) |
| Nemotron 3.5 Super VL 120B-A12B | DGX Spark (GB10, 128GB): Nemotron 3.5 Super VL 120B and Bolt 74B, launch-week runs | DGX Spark (GB10) (128 GB) | NVFP4, no speculative decoding, 256-token output | vLLM | 19.06 | |
| Qwen3.6 35B-A3B | MacBook Pro M5 Max 128 GB: Qwen3.6 35B-A3B at 271 tok/s (mlx-serve + MTP), Gemma 4 26B-A4B at 166 | Apple M5 Max (unified) (128 GB) | 4-bit MLX (mlx-serve build) | mlx-serve 26.10.2 + MTP | 271 | ⚡ spec-decode (MTP) |
| Qwen3.6 35B-A3B | MacBook Pro M5 Max 128 GB: Qwen3.6 35B-A3B at 271 tok/s (mlx-serve + MTP), Gemma 4 26B-A4B at 166 | Apple M5 Max (unified) (128 GB) | 4-bit MLX | Rapid-MLX 0.16.1 + MTP | 240 | ⚡ spec-decode (MTP) |
| Qwen3.6 35B-A3B | DGX Spark (GB10): Qwen3.6 35B-A3B NVFP4 at 113 tok/s on vLLM with MTP | DGX Spark (GB10) (128 GB) | NVFP4 | vLLM 0.30 + MTP | 113 | ⚡ spec-decode (MTP) |
| Qwen3.6 35B-A3B | MacBook Pro M5 Max 128 GB: Qwen3.6 35B-A3B at 271 tok/s (mlx-serve + MTP), Gemma 4 26B-A4B at 166 | Apple M5 Max (unified) (128 GB) | 4-bit MLX | mlx_lm.server 0.32.2 | 107 | |
| Qwen3.6 35B-A3B | DGX Spark (GB10, 128 GB): 23 models on vLLM at 128K context, Qwen3.6 35B-A3B up to 102 tok/s | DGX Spark (GB10) (128 GB) | NVFP4 + MTP (k=2) | vLLM 0.31.0, 128K ctx, 1 user | 102.3 | ⚡ spec-decode (MTP) |
| Qwen3.6 35B-A3B | Minisforum MS-S1 MAX (Strix Halo, 128 GB): Qwen3.6 35B-A3B at 62 tok/s on llama.cpp Vulkan | Strix Halo (Radeon 8060S) (128 GB) | Q4_K_M (Unsloth) | llama.cpp Vulkan + MTP | 93 | ⚡ spec-decode (MTP) |
| Qwen3.6 35B-A3B | HP OmniBook Ultra 16 (RTX Spark N1X, 128GB) laptop: local AI benchmarks | RTX Spark N1X (128 GB) | Ollama default, 64K context, Codex CLI coding agent | Ollama | 89.5 | |
| Qwen3.6 35B-A3B | DGX Spark (GB10): Qwen3.6 35B-A3B NVFP4 at 113 tok/s on vLLM with MTP | DGX Spark (GB10) (128 GB) | NVFP4 | vLLM 0.30 + MTP | 86 | ⚡ spec-decode (MTP) |
| Qwen3.6 35B-A3B | DGX Spark (GB10, 128 GB): 23 models on vLLM at 128K context, Qwen3.6 35B-A3B up to 102 tok/s | DGX Spark (GB10) (128 GB) | NVFP4, no speculative decoding | vLLM 0.31.0, 128K ctx, 1 user | 78.1 | |
| Qwen3.6 35B-A3B | Minisforum MS-S1 MAX (Strix Halo, 128 GB): Qwen3.6 35B-A3B at 62 tok/s on llama.cpp Vulkan | Strix Halo (Radeon 8060S) (128 GB) | Q4_K_M (Unsloth) | llama.cpp Vulkan | 62 | |
| Qwen3.6 35B-A3B | Strix Halo (Corsair AI Workstation 300, 128 GB): ROCm vs Vulkan on Qwen3.8 27B and Qwen3.6 35B-A3B | Strix Halo (Radeon 8060S) (128 GB) | Q8_0 + MTP (draft-n-max 3), coding workload | llama.cpp ROCm | 60.92 | ⚡ spec-decode (MTP) |
| Qwen3.6 35B-A3B | HP ZBook Ultra G1a (Strix Halo laptop, 128 GB): Qwen3.6 35B-A3B at 59.4 tok/s, gpt-oss 120B at 44.7 | Strix Halo (Radeon 8060S) (128 GB) | Q4_K_M | llama.cpp b11213 Vulkan | 59.4 | |
| Qwen3.6 35B-A3B | HP ZBook Ultra G1a (Strix Halo laptop, 128 GB): Qwen3.6 35B-A3B at 59.4 tok/s, gpt-oss 120B at 44.7 | Strix Halo (Radeon 8060S) (128 GB) | 4-bit | Ollama 0.34.0 | 54.4 | |
| Qwen3.6 35B-A3B | Strix Halo (Corsair AI Workstation 300, 128 GB): ROCm vs Vulkan on Qwen3.8 27B and Qwen3.6 35B-A3B | Strix Halo (Radeon 8060S) (128 GB) | Q8_0, llama-bench tg128 | llama.cpp Vulkan | 54.22 | |
| Qwen3.6 35B-A3B | Minisforum MS-S1 MAX (Strix Halo, 128 GB): Qwen3.6 35B-A3B at 62 tok/s on llama.cpp Vulkan | Strix Halo (Radeon 8060S) (128 GB) | Q8_0 | llama.cpp Vulkan | 54 | |
| Qwen3.6 35B-A3B | Minisforum MS-S1 MAX (Strix Halo, 128 GB): Qwen3.6 35B-A3B at 62 tok/s on llama.cpp Vulkan | Strix Halo (Radeon 8060S) (128 GB) | Q4_K_M (Unsloth) | llama.cpp ROCm | 51 | |
| Qwen3.6 35B-A3B | Strix Halo (Corsair AI Workstation 300, 128 GB): ROCm vs Vulkan on Qwen3.8 27B and Qwen3.6 35B-A3B | Strix Halo (Radeon 8060S) (128 GB) | Q8_0, llama-bench tg128 | llama.cpp ROCm | 47.47 | |
| Qwen3.8 27B | Apple M5 Max (40-core GPU) 128 GB running Qwen3.8 27B with MTP drafter | Apple M5 Max (40-core GPU, unified) (128 GB) | 4-bit MLX + Qwen3.8-27B-MTP-4bit drafter | oMLX v0.6.0, MTP enabled | 45.3 | ⚡ spec-decode (MTP)prefill 963.4 tok/scontext: 1K (45.8 tok/s at 4K)📏 oMLX community benchmark upload, peak memory 19.1 GB |
| Qwen3.8 27B | MacBook Pro M5 Max 128 GB: Qwen3.6 35B-A3B at 271 tok/s (mlx-serve + MTP), Gemma 4 26B-A4B at 166 | Apple M5 Max (unified) (128 GB) | 4-bit MLX | same server, no speculation | 31 | |
| Qwen3.8 27B | Mac Studio M5 Ultra 512GB: Qwen3.8 27B and Mistral Medium 3.5 (Apple MLX benchmark) | Apple M5 Ultra (512 GB) | BF16, 8,192-token context | MLX (mlx_lm.benchmark) | 19.69 | |
| Qwen3.8 27B | DGX Spark (GB10, 128 GB): 23 models on vLLM at 128K context, Qwen3.6 35B-A3B up to 102 tok/s | DGX Spark (GB10) (128 GB) | + MTP (k=2) | vLLM 0.31.0, 128K ctx, 1 user | 19.0 | ⚡ spec-decode (MTP) |
| Qwen3.8 27B | Strix Halo (Ryzen AI Max+ 395, 128 GB): Qwen3.8 27B Q6_K, 9.6 to 18 tok/s with MTP | Strix Halo (Radeon 8060S) (128 GB) | Q6_K + MTP (draft-n-max 2) | llama.cpp Vulkan (RADV) | 18.01 | ⚡ spec-decode (MTP) |
| Qwen3.8 27B | Strix Halo (Corsair AI Workstation 300, 128 GB): ROCm vs Vulkan on Qwen3.8 27B and Qwen3.6 35B-A3B | Strix Halo (Radeon 8060S) (128 GB) | Q8_0 + MTP (draft-n-max 3), coding workload, reasoning on | llama.cpp ROCm | 15.67 | ⚡ spec-decode (MTP) |
| Qwen3.8 27B | Strix Halo (Ryzen AI Max+ 395, 128 GB): Qwen3.8 27B Q6_K, 9.6 to 18 tok/s with MTP | Strix Halo (Radeon 8060S) (128 GB) | Q6_K + MTP (draft-n-max 5) | llama.cpp Vulkan (RADV) | 15.16 | ⚡ spec-decode (MTP) |
| Qwen3.8 27B | Strix Halo (Corsair AI Workstation 300, 128 GB): ROCm vs Vulkan on Qwen3.8 27B and Qwen3.6 35B-A3B | Strix Halo (Radeon 8060S) (128 GB) | Q8_0 + MTP (draft-n-max 3), coding workload, reasoning on | llama.cpp Vulkan | 13.19 | ⚡ spec-decode (MTP) |
| Qwen3.8 27B | Strix Halo mini PC (Ryzen AI Max+ 395, 128 GB): Qwen3.8 27B at 12.6 tok/s Q4_K_M, 7.7 Q8_0 | Strix Halo (Radeon 8060S) (128 GB) | Q4_K_M | llama.cpp/Ollama, ROCm | 12.6 | |
| Qwen3.8 27B | Mac Studio M3 Ultra 512GB: Qwen3.8 27B and Mistral Medium 3.5 (Apple MLX benchmark) | Apple M3 Ultra (512 GB) | BF16, 8,192-token context | MLX (mlx_lm.benchmark) | 12.46 | |
| Qwen3.8 27B | DGX Spark (GB10, 128 GB): 23 models on vLLM at 128K context, Qwen3.6 35B-A3B up to 102 tok/s | DGX Spark (GB10) (128 GB) | no speculative decoding | vLLM 0.31.0, 128K ctx, 1 user | 11.0 | |
| Qwen3.8 27B | Strix Halo (Ryzen AI Max+ 395, 128 GB): Qwen3.8 27B Q6_K, 9.6 to 18 tok/s with MTP | Strix Halo (Radeon 8060S) (128 GB) | Q6_K, no MTP | llama.cpp Vulkan (RADV) | 9.59 | |
| Qwen3.8 27B | Minisforum MS-S1 MAX (Ryzen AI Max+ PRO 495, 192 GB): gpt-oss 120B at 55.8 tok/s | AMD Ryzen AI Max+ PRO 495 (Radeon 8065S iGPU, 40 CUs) (192 GB) | Q8 (author: "about 8 tok/s") | llama.cpp Vulkan | 8 | |
| Qwen3.8 27B | Strix Halo (Corsair AI Workstation 300, 128 GB): ROCm vs Vulkan on Qwen3.8 27B and Qwen3.6 35B-A3B | Strix Halo (Radeon 8060S) (128 GB) | Q8_0, llama-bench tg128 | llama.cpp Vulkan | 7.77 | |
| Qwen3.8 27B | Strix Halo (Corsair AI Workstation 300, 128 GB): ROCm vs Vulkan on Qwen3.8 27B and Qwen3.6 35B-A3B | Strix Halo (Radeon 8060S) (128 GB) | Q8_0, llama-bench tg128 | llama.cpp ROCm | 7.76 | |
| Qwen3.8 27B | Strix Halo mini PC (Ryzen AI Max+ 395, 128 GB): Qwen3.8 27B at 12.6 tok/s Q4_K_M, 7.7 Q8_0 | Strix Halo (Radeon 8060S) (128 GB) | Q8_0, --reasoning off | llama.cpp/Ollama, ROCm | 7.7 | |
| Qwen3.8 Flash-Next | DGX Spark (GB10, 128 GB): 23 models on vLLM at 128K context, Qwen3.6 35B-A3B up to 102 tok/s | DGX Spark (GB10) (128 GB) | + MTP (k=2) | patched vLLM (qwen38-flash-dgx), 128K ctx, 1 user | 26.8 | ⚡ spec-decode (MTP) |
| Step 5 Preview | 4x DGX Spark (GB10) running Step 5 Preview with tensor parallel over RoCE | 4x DGX Spark (GB10) (512 GB) | EXL3 (experts 4.01 bpw, 341.4 GB total) | vLLM, TP=4 over RoCE, MTP (2 draft tokens) | 25 | ⚡ spec-decode (MTP)prefill 1,365 tok/scontext: 8K📏 ~25–30 tok/s single stream (published range); prefill 1,365–1,565 tok/s warm at 8K; 66.5 tok/s aggregate with 4 streams |
| gpt-oss 120B | Mac Studio M2 Ultra 76-core GPU, 192 GB | Apple M2 Ultra (192 GB) | MXFP4 | llama.cpp | 79.68 | |
| gpt-oss 120B | Minisforum MS-S1 MAX (Ryzen AI Max+ PRO 495, 192 GB): gpt-oss 120B at 55.8 tok/s | AMD Ryzen AI Max+ PRO 495 (Radeon 8065S iGPU, 40 CUs) (192 GB) | MXFP4 | llama.cpp Vulkan | 55.8 | |
| gpt-oss 120B | Minisforum MS-S1 MAX (Ryzen AI Max+ PRO 495, 192 GB): gpt-oss 120B at 55.8 tok/s | AMD Ryzen AI Max+ PRO 495 (Radeon 8065S iGPU, 40 CUs) (192 GB) | MXFP4, 32K context | llama.cpp Vulkan | 45.1 | |
| gpt-oss 120B | HP ZBook Ultra G1a (Strix Halo laptop, 128 GB): Qwen3.6 35B-A3B at 59.4 tok/s, gpt-oss 120B at 44.7 | Strix Halo (Radeon 8060S) (128 GB) | MXFP4 | llama.cpp b11213 Vulkan | 44.7 | |
| gpt-oss 120B | DGX Spark (GB10, 128 GB): 23 models on vLLM at 128K context, Qwen3.6 35B-A3B up to 102 tok/s | DGX Spark (GB10) (128 GB) | vLLM 0.31.0, 128K ctx, 1 user | 44.4 | ||
| gpt-oss 20B | Mac Studio M2 Ultra 76-core GPU, 192 GB | Apple M2 Ultra (192 GB) | MXFP4 | llama.cpp | 116.08 | |
| gpt-oss 20B | Mac Studio M3 Ultra 512GB (Geerling test) | Apple M3 Ultra (512 GB) | MXFP4 | llama.cpp (Metal, llama-bench) | 115.52 | |
| gpt-oss 20B | Mac Studio M3 Ultra 512GB (Geerling test) | Apple M3 Ultra (512 GB) | llama.cpp | 115.29 | ||
| gpt-oss 20B | Mac Studio M1 Ultra (64-core GPU) 128GB | Apple M1 Ultra (128 GB) | llama.cpp | 100.43 | ||
| gpt-oss 20B | DGX Spark (GB10, 128 GB): 23 models on vLLM at 128K context, Qwen3.6 35B-A3B up to 102 tok/s | DGX Spark (GB10) (128 GB) | vLLM 0.31.0, 128K ctx, 1 user | 96.2 | ||
| gpt-oss 20B | HP OmniBook Ultra 16 (RTX Spark N1X, 128GB) laptop: local AI benchmarks | RTX Spark N1X (128 GB) | Ollama default, 64K context, Codex CLI coding agent | Ollama | 58.0 | |
| DeepSeek R1 671B legacy | Mac Studio M3 Ultra 512GB (Geerling test) | Apple M3 Ultra (512 GB) | llama.cpp | 19.89 | ||
| DeepSeek R1 671B legacy | Mac Studio M3 Ultra, 512 GB ⚠ unchecked | Apple M3 Ultra (512 GB) | Q4 | MLX | 18.43 | |
| DeepSeek R1 Distill 14B legacy | Mac Studio M3 Ultra 512GB (Geerling test) | Apple M3 Ultra (512 GB) | llama.cpp | 51.85 | ||
| DeepSeek R1 Distill 14B legacy | Mac Studio M1 Ultra (64-core GPU) 128GB | Apple M1 Ultra (128 GB) | llama.cpp | 41.51 | ||
| DeepSeek V4 Flash legacy | 8x RTX 3090 serving DeepSeek V4 Flash (LimeChain vLLM kit) | 8x RTX 3090 (192 GB) | see source (FP8 KV cache) | vLLM LimeChain SM86 overlay, 1 concurrent request | 95.52 | 📏 Per-session end-to-end p50 at concurrency 1, 128 generated tokens |
| DeepSeek V4 Flash legacy | Minisforum MS-S1 MAX (Ryzen AI Max+ PRO 495, 192 GB): gpt-oss 120B at 55.8 tok/s | AMD Ryzen AI Max+ PRO 495 (Radeon 8065S iGPU, 40 CUs) (192 GB) | IQ3_S | llama.cpp Vulkan | 17.4 | |
| DeepSeek V4 Flash legacy | Minisforum MS-S1 MAX (Ryzen AI Max+ PRO 495, 192 GB): gpt-oss 120B at 55.8 tok/s | AMD Ryzen AI Max+ PRO 495 (Radeon 8065S iGPU, 40 CUs) (192 GB) | IQ3_S, 32K context | llama.cpp Vulkan | 12.2 | |
| GLM-5.2 legacy | 4x DGX Spark cluster running GLM-5.2 (unpruned) | 4x DGX Spark (GB10) (512 GB) | QuantTrio Int4-Int8Mix | vLLM TP4 + MTP, single-stream mean | 32.5 | ⚡ spec-decode (MTP)context: 200K max📏 Single-stream mean (source config of 2026-07-17) |
| Llama 2 7B legacy | Mac Studio M2 Ultra 76-core GPU, 192 GB | Apple M2 Ultra (192 GB) | Q4_0 | llama.cpp Metal (FA, Aug 2026 build) | 125.21 | |
| Llama 3.1 405B legacy | Mac Studio M3 Ultra 512GB (Geerling test) | Apple M3 Ultra (512 GB) | llama.cpp | 1.96 | ||
| Llama 3.1 70B legacy | Mac Studio M3 Ultra 512GB (Geerling test) | Apple M3 Ultra (512 GB) | llama.cpp | 14.08 | ||
| Llama 3.1 70B legacy | Mac Studio M1 Ultra (64-core GPU) 128GB | Apple M1 Ultra (128 GB) | llama.cpp | 9.84 | ||
| Llama 3.1 8B legacy | HP OmniBook Ultra 16 (RTX Spark N1X, 128GB) laptop: local AI benchmarks | RTX Spark N1X (128 GB) | MLPerf Client v2.0 (geomean) | llama.cpp CUDA | 56.5 | |
| Llama 3.1 8B legacy | DGX Spark (GB10, 128 GB): 23 models on vLLM at 128K context, Qwen3.6 35B-A3B up to 102 tok/s | DGX Spark (GB10) (128 GB) | NVIDIA NIM 2.0.9, 128K ctx, 1 user | 26.4 | ||
| Llama 3.2 3B legacy | Mac Studio M3 Ultra 512GB (Geerling test) | Apple M3 Ultra (512 GB) | llama.cpp | 154.6 | ||
| Llama 3.2 3B legacy | Mac Studio M1 Ultra (64-core GPU) 128GB | Apple M1 Ultra (128 GB) | llama.cpp | 127.6 | ||
| MiniMax M2.7 legacy | Minisforum MS-S1 MAX (Ryzen AI Max+ PRO 495, 192 GB): gpt-oss 120B at 55.8 tok/s | AMD Ryzen AI Max+ PRO 495 (Radeon 8065S iGPU, 40 CUs) (192 GB) | IQ4_XS | llama.cpp Vulkan | 29.5 | |
| MiniMax M2.7 legacy | Minisforum MS-S1 MAX (Ryzen AI Max+ PRO 495, 192 GB): gpt-oss 120B at 55.8 tok/s | AMD Ryzen AI Max+ PRO 495 (Radeon 8065S iGPU, 40 CUs) (192 GB) | IQ4_XS, 32K context | llama.cpp Vulkan | 17.3 | |
| MiniMax M3 legacy | Mac Studio M3 Ultra 512 GB running MiniMax M3 (4-bit MLX) | Apple M3 Ultra (512 GB) | 4-bit MLX | oMLX v0.4.4rc2 | 28.9 | prefill 332.6 tok/scontext: 1K (20.8 tok/s at 4K)📏 oMLX community benchmark upload, peak memory 226.6 GB |
| Nemotron 3 Super 120B-A12B legacy | DGX Spark (GB10, 128 GB): 23 models on vLLM at 128K context, Qwen3.6 35B-A3B up to 102 tok/s | DGX Spark (GB10) (128 GB) | + MTP (k=3) | vLLM 0.31.0, 128K ctx, 1 user | 23.0 | ⚡ spec-decode (MTP) |
| Nemotron 3 Super 120B-A12B legacy | DGX Spark (GB10, 128 GB): 23 models on vLLM at 128K context, Qwen3.6 35B-A3B up to 102 tok/s | DGX Spark (GB10) (128 GB) | no speculative decoding | vLLM 0.31.0, 128K ctx, 1 user | 16.1 | |
| Qwen3.6 27B legacy | DGX Spark with Qwen3.6 27B NVFP4 + DFlash speculative decoding | DGX Spark (GB10) (128 GB) | NVFP4 | vLLM + DFlash spec decoding | 32.83 | ⚡ spec-decode (DFlash)📏 Median decode, Docker reproduction |
| Qwen3.6 27B legacy | HP ZBook Ultra G1a (Strix Halo laptop, 128 GB): Qwen3.6 35B-A3B at 59.4 tok/s, gpt-oss 120B at 44.7 | Strix Halo (Radeon 8060S) (128 GB) | 4-bit | Ollama 0.34.0 | 11.4 | |
| Qwen3.6 27B legacy | DGX Spark with Qwen3.6 27B NVFP4 + DFlash speculative decoding | DGX Spark (GB10) (128 GB) | FP8 | vLLM autoregressive baseline | 7.85 | 📏 Autoregressive baseline, same reproduction |
What fits by memory math
Q4_K_M weights (4.8944 bits per weight, per the llama.cpp quantize table) that fit in 128 GB with ~5% headroom, before KV cache: Qwen3.8 Flash-Next ≈110 GB, Mistral Medium 3.5 128B ≈78 GB, Ornith 1.5 122B ≈77 GB, Qwen3.5 122B legacy ≈75 GB, Nemotron 3 Super 120B-A12B legacy ≈73 GB, Nemotron 3.5 Super VL 120B-A12B ≈73 GB, gpt-oss 120B ≈72 GB, Kolibri-1 ≈48 GB, Nemotron 3.5 Bolt 74B-A9B ≈45 GB, Llama 3.3 70B legacy ≈43 GB, Flint 40B-A2.5B ≈24 GB, Ornith 1.5 35B-A3B ≈22 GB, Qwen3.6 35B-A3B ≈22 GB, Gemma 4 31B ≈19 GB, Nemotron 3.5 Lightning 30B-A3B ≈18 GB, Qwen3.6 27B legacy ≈17 GB, Qwen3.8 27B ≈17 GB, Gemma 4 26B-A4B ≈15 GB, Devstral Small 2 24B ≈15 GB, LFM2.5 24B-A2B ≈15 GB, gpt-oss 20B ≈13 GB, Mellum 2.1 12B-A2.5B ≈7 GB, Gemma 4 12B ≈7 GB, Qwen3.5 9B ≈5 GB, Llama 3.1 8B legacy ≈5 GB.
FAQ
Which local LLMs have been measured on 128 GB or more of VRAM?
Llama 2 7B, gpt-oss 20B, gpt-oss 120B, DeepSeek R1 671B, Llama 3.1 70B, Llama 3.1 405B, DeepSeek R1 Distill 14B, Llama 3.2 3B, Muse Spark Open 816B-A42B, Qwen3.8 27B, Mistral Medium 3.5 128B, Qwen3.6 35B-A3B, Llama 3.1 8B, Nemotron 3.5 Super VL 120B-A12B, Nemotron 3.5 Bolt 74B-A9B, Nemotron 3.5 Lightning 30B-A3B, Nemotron 3 Nano Omni 30B, Gemma 4 26B-A4B, Qwen3.8 Flash-Next, Nemotron 3 Super 120B-A12B, MiniMax M2.7, DeepSeek V4 Flash, Qwen3.6 27B, Gemma 4 31B, GLM-5.2, GLM-5.3 Flash, MiniMax M3, DeepSeek V4.1 Flash, MiniMax M3.1 Flash, Kolibri-1, Step 5 Preview (93 results).
What fits in 128 GB or more of VRAM?
By memory math at Q4_K_M (4.8944 bits per weight, llama.cpp), weights only, before KV cache and context: Qwen3.8 Flash-Next (≈110 GB), Mistral Medium 3.5 128B (≈78 GB), Ornith 1.5 122B (≈77 GB), Qwen3.5 122B (≈75 GB), Nemotron 3 Super 120B-A12B (≈73 GB), Nemotron 3.5 Super VL 120B-A12B (≈73 GB), gpt-oss 120B (≈72 GB), Kolibri-1 (≈48 GB). Leave a few GB for context.
Are speculative-decoding results comparable?
Not 1:1. MTP, DFlash and similar methods draft tokens that the main model verifies, which can double tok/s on predictable text. This page lists them in a separate column; tick the box to hide them in the full table.
Not enough VRAM? Rent before you buy. Run bigger models on a rented 24–96 GB cloud GPU by the hour and see the real speed before spending €1,000+. Try it on Runpod
Referral link: if you sign up with Google and add $10, Runpod gives you and RigBase bonus credit (terms). It never affects our data.
Referral link: if you sign up with Google and add $10, Runpod gives you and RigBase bonus credit (terms). It never affects our data.