Orkun Soylu

1 min read

MTP speculative decoding @Ollama: 20 → 42 tok/s

Turning on the model's built-in multi-token-prediction head more than doubled generation speed without changing quality. Four draft tokens is the sweet spot, and the gain depends a lot on content — code accepts drafts far more often than Turkish prose.

Tested
Setup

Radeon Pro W7800 48 GB — bare-metal workstationBare metal, Linux, ROCm (Ollama) · AMD Ryzen 5 7600 (6C/12T, AVX-512) · 64 GB DDR5-6400

Runtime
Ollama 0.34.0 (llama-server --spec-type draft-mtp)
Models

qwen3.8:27b-q8_0-mtpQ8_0 + MTPtested

qwen3.8:27bQ8_0baseline

Verdict

The single biggest speed-up of this whole series: +114% from one Modelfile line. Use draft_num_predict 4, and skip DFlash-style external drafts if you write anything other than English.

Works

  • 19.8 → 42.3 tok/s with draft_num_predict 4
  • One Modelfile parameter when the GGUF carries the nextn head
  • Thinking on or off makes no difference to the gain

Doesn’t

  • Ollama drops to a single slot with a draft model (n_seq_max=1)
  • Turkish prose accepts only 2.0–2.4 tokens per step (28–35 tok/s) vs 3.4–3.6 for code (45–48)
  • External DFlash2 draft: 57 tok/s on code but 21 on Turkish — rejected
Without MTP
19.8tok/s
With MTP, n = 4
42.3tok/s
Speed-up
2.14×
VRAM at 262K
38 → 30GB

Draft length sweep (Q8_0, temperature 1)

higher is better
No MTP
19.8 tok/sGeneration · 19.8 tok/s
MTP n = 2
38.6 tok/sGeneration · 38.6 tok/s
MTP n = 3
40.6 tok/sGeneration · 40.6 tok/s
MTP n = 4 (chosen)
42.3 tok/sGeneration · 42.3 tok/s
MTP n = 5
40.5 tok/sGeneration · 40.5 tok/s
DFlash2 external draft, n = 7
38.2 tok/sGeneration · 38.2 tok/s
Data
Generation
No MTP19.8 tok/s
MTP n = 238.6 tok/s
MTP n = 340.6 tok/s
MTP n = 4 (chosen)42.3 tok/s
MTP n = 540.5 tok/s
DFlash2 external draft, n = 738.2 tok/s

Speed depends on the content

higher is better

MTP n = 4, midpoint of the measured range

  • MTP n = 4
  • DFlash2 draft
Code
46.5 tok/sMTP n = 4 · 46.5 tok/s
57.0 tok/sDFlash2 draft · 57.0 tok/s
Turkish prose
31.5 tok/sMTP n = 4 · 31.5 tok/s
21.0 tok/sDFlash2 draft · 21.0 tok/s
Data
MTP n = 4DFlash2 draft
Code46.5 tok/s57.0 tok/s
Turkish prose31.5 tok/s21.0 tok/s

How it works in Ollama

Ollama 0.34 passes --spec-type draft-mtp to llama-server. If the GGUF carries the model’s own multi-token-prediction weights (blk.N.nextn.*), all it takes is one Modelfile line:

PARAMETER draft_num_predict 4

You can also point it at a separate draft model with DRAFT <gguf> instead.

The small MTP head guesses several tokens ahead, and the full model checks them all in a single pass. Every accepted token is free speed; every rejected one costs a little. The output quality doesn’t change — the big model still has the final say on every token.

Results

Setting (Q8_0, temperature 1) Generation
No MTP 19.8 tok/s
n = 2 38.6
n = 3 40.6
n = 4 (chosen) 42.3
n = 5 40.5
DFlash2 external draft (n = 7) 38.2 — 57 on code, 21 on Turkish → rejected

With a draft active, Ollama drops to a single slot (n_seq_max=1), and VRAM at 262K actually falls from 38 to 30 GB.

One number would mislead you

The speed depends on how predictable the text is:

  • Code: 45–48 tok/s (3.4–3.6 tokens accepted per step)
  • Turkish prose: 28–35 tok/s (2.0–2.4 per step)
  • Thinking on or off: no difference.

So always measure MTP on your own language and your own workload.

What happened next

Q8_0 + MTP (qwen3.8:27b-q8_0-mtp, 262K, 39 GiB) became my homelab’s default, and the baseline I compared every later model on this site against.