Orkun Soylu

3 min read

Qwen3.6 35B-A3B MoE vs 27B @26k token prompt

On my 48 GB W7800 the mixture-of-experts model generates 3.4× faster than the dense 27B. Filling a quarter of the context barely dents generation speed — almost all of the wait is prefill.

Tested
Setup

Radeon Pro W7800 48 GB — bare-metal workstationBare metal, Linux, ROCm (Ollama) · AMD Ryzen 5 7600 (6C/12T, AVX-512) · 64 GB DDR5-6400

Radeon RX 7900 XTX 24 GB — bare metalBare metal, Ollama (ROCm)

Runtime
Ollama 0.31.1 (ROCm), flash attention on, q8_0 KV cache
Models

qwen3.6:35b-a3bQ8_0tested

qwen3.6:27bQ8_0tested

qwen3.6:27b (7900 XTX)Q4_K_Mreference

Verdict

MoE wins on speed, and not by a little (66 vs 20 tok/s). Context you merely configure costs only VRAM; context you actually fill costs mostly prefill time, not generation speed — at least up to ~26K tokens.

Works

  • 35B-A3B MoE: 66 tok/s short-prompt generation and 1,711 tok/s prefill on a single W7800
  • Generation drops only 14–16% with a 26K-token prompt
  • Raising the configured context from 32K to 64K costs ~1.3 GB of VRAM and no speed

Doesn’t

  • Dense 27B Q8_0 waits 43.5 s before the first token on a 25K prompt
  • The 7900 XTX data point used a different quant (Q4_K_M), so it cannot isolate the GPU
MoE generation, short prompt
66.3tok/s
Dense generation, short prompt
19.6tok/s
Prefill 26K, MoE vs dense
15.5 / 43.5s
Slowdown at ~26K context
−16 / −14%

Generation speed

higher is better

Ollama /api/generate eval rate, short (28-token) vs ~26K-token prompt

  • Short prompt
  • ~26K-token prompt
35B-A3B MoE Q8_0 · W7800
66.3 tok/sShort prompt · 66.3 tok/s
55.9 tok/s~26K-token prompt · 55.9 tok/s
27B dense Q8_0 · W7800
19.6 tok/sShort prompt · 19.6 tok/s
16.8 tok/s~26K-token prompt · 16.8 tok/s
27B dense Q4_K_M · 7900 XTX
33.6 tok/sShort prompt · 33.6 tok/s
31.1 tok/s~26K-token prompt · 31.1 tok/s
Data
Short prompt~26K-token prompt
35B-A3B MoE Q8_0 · W780066.3 tok/s55.9 tok/s
27B dense Q8_0 · W780019.6 tok/s16.8 tok/s
27B dense Q4_K_M · 7900 XTX33.6 tok/s31.1 tok/s

Prefill (prompt processing) speed

higher is better

~25–26K-token prompt

35B-A3B MoE Q8_0 · W7800
1,711.3 tok/sPrefill · 1,711.3 tok/s
27B dense Q8_0 · W7800
582.6 tok/sPrefill · 582.6 tok/s
27B dense Q4_K_M · 7900 XTX
768.0 tok/sPrefill · 768.0 tok/s
Data
Prefill
35B-A3B MoE Q8_0 · W78001,711.3 tok/s
27B dense Q8_0 · W7800582.6 tok/s
27B dense Q4_K_M · 7900 XTX768.0 tok/s

How I measured

  • Short prompt: a 28-token question (“What is the capital of Turkey?”).
  • Long prompt: ~99,000 characters of repeated filler text plus a short instruction. That tokenises to 25,344 (dense) / 26,444 (MoE) tokens — roughly 40% of the 64K context I configured for the test.
  • The numbers come straight from Ollama’s own prompt_eval_count/duration (prefill) and eval_count/duration (generation) fields in the /api/generate response.
  • Server settings: OLLAMA_CONTEXT_LENGTH=65536, OLLAMA_FLASH_ATTENTION=1, OLLAMA_KV_CACHE_TYPE=q8_0.

Results — W7800 48 GB

qwen3.6:35b-a3b Q8_0 (MoE, 36B total / ~3B active)

Short prompt (28 tok) Long prompt (26,444 tok)
Prefill 299.7 tok/s 1,711.3 tok/s
Prefill time 0.09 s 15.45 s
Generation 66.3 tok/s 55.9 tok/s
Total time instant 22.4 s

qwen3.6:27b Q8_0 (dense, 27.8B — every parameter works on every token)

Short prompt (28 tok) Long prompt (25,344 tok)
Prefill 145.7 tok/s 582.6 tok/s
Prefill time 0.19 s 43.50 s
Generation 19.6 tok/s 16.8 tok/s
Total time instant 66.0 s

Results — RX 7900 XTX 24 GB

Only qwen3.6:27b-q4_K_M fits on this card (Q8_0 is ~30 GB), and I ran it at the default 32K context.

Short prompt (28 tok) Long prompt (25,344 tok)
Prefill 188.5 tok/s 768.0 tok/s
Prefill time 0.15 s 33.00 s
Generation 33.6 tok/s 31.1 tok/s
VRAM 18.7 GB 18.7 GB

The 7900 XTX comes out at ~1.7× the W7800’s dense Q8_0 — but the quantization is different, so most of that gap is Q4 vs Q8, not the GPU. I did the clean, single-variable GPU comparison six weeks later: W7800 vs RX 7900 XTX.

Is 33 tok/s on the 7900 XTX a fault?

It looked low to me. The card has ~960 GB/s and the model is 18.7 GB, which puts the bandwidth ceiling around 51 tok/s — so 33.6 is only ~65% efficiency. I checked two suspects:

  1. Missing flash attention / q8_0 KV? I added both and restarted: 32.6–32.7 tok/s, no change. Not surprising — the KV cache is tiny on a short prompt.
  2. Clock throttling? I watched rocm-smi live during inference: ~2,600 MHz, 96% busy, ~303 W. Fully boosted.

So 33 tok/s really is the ceiling for Q4_K_M here. The likely reason, which I confirmed later: unpacking K-quant weights costs compute that the bandwidth arithmetic ignores, and RDNA3’s consumer kernels pay a lot for it.

What I learned

  1. Configured context is (almost) free. Going from 32K to 64K reserves more KV cache (~1.3 GB for the 27B), but short-prompt speed doesn’t move at all.
  2. Filled context mostly costs prefill. At ~26K tokens generation drops only 16% (MoE) / 14% (dense). What you actually feel is the wait for the first token: 15.5 s on the MoE, 43.5 s on the dense model.
  3. If you want speed, take the MoE. Only ~3B parameters are computed per token, so the 35B-A3B generates ~3.4× faster and prefills 2–3× faster. It doesn’t save VRAM — all the experts stay loaded — but on a 48 GB card that isn’t the constraint.

All of this holds at ~26K tokens. At 168K the picture changes sharply — see 168K-token context on a dense 27B.