Qwen3.8-27B Q8_0 vs Q4_K_M
Dropping from Q8_0 to Q4_K_M on my W7800 bought 29% faster generation and 12 GB of VRAM — far less than the bandwidth maths promised — and made long-prompt prefill 7% slower.
- Tested
- Setup
Radeon Pro W7800 48 GB — bare-metal workstationBare metal, Linux, ROCm (Ollama) · AMD Ryzen 5 7600 (6C/12T, AVX-512) · 64 GB DDR5-6400
- Runtime
- Ollama 0.32.14, ROCm 7.2.4, flash attention on, q8_0 KV cache, 262K context
- Models
qwen3.8:27bQ8_0baseline
qwen3.8:27bQ4_K_Mtested
Verdict
Q4_K_M is worth it for VRAM, not for speed: generation gains 24–29%, prefill gets slightly worse, and the advantage shrinks as the context fills. 20 tok/s on Q8_0 is simply this card's ceiling, not a fault.
Works
- −12 GB VRAM (26 GB vs 38 GB at 262K)
- +29% generation on short prompts, +24% at 31K tokens
Doesn’t
- Predicted ~35 tok/s, measured 25.7 — bandwidth efficiency falls from 69% to 53%
- Prefill is 7% slower than Q8_0 (498 vs 534 tok/s)
- Loses more speed as context fills (−19% vs −16%)
- Generation, Q8_0 → Q4_K_M
- 19.9 → 25.7tok/s
- Bandwidth efficiency
- 69% → 53%
- Prefill 31K tokens
- 534 → 498tok/s
- VRAM at 262K
- 38 → 26GB
Generation speed
higher is better- Q8_0
- Q4_K_M
Short prompt (21 tok)
Long prompt (31,073 tok)
Theoretical ceiling (864 GB/s ÷ weights)
Data
| Q8_0 | Q4_K_M | |
|---|---|---|
| Short prompt (21 tok) | 19.85 tok/s | 25.68 tok/s |
| Long prompt (31,073 tok) | 16.77 tok/s | 20.75 tok/s |
| Theoretical ceiling (864 GB/s ÷ weights) | 28.80 tok/s | 48.70 tok/s |
Prefill, 31,073-token prompt
higher is better- Q8_0
- Q4_K_M
Prefill
Data
| Q8_0 | Q4_K_M | |
|---|---|---|
| Prefill | 533.6 tok/s | 498.0 tok/s |
Why I measured
In daily use qwen3.8:27b-q8_0 felt slow at ~20 tok/s. Was something wrong, and would Q4_K_M fix it?
How I measured
- Short prompt: 21 tokens (“write me a Python code example”), 2–3 repetitions.
- Long prompt: 99,000 characters of Turkish filler plus an instruction — 31,073 tokens (Turkish tokenises worse than English).
/api/chat,stream: false,think: false. I unloaded both variants withkeep_alive: 0before every measurement (30 + 18 GB don’t fit together).- Both use a
q8_0KV cache. The KV type doesn’t depend on the weight quantization, so at 31K tokens both models move the same number of KV bytes. Keep that in mind for point 5.
Results
| Q8_0 | Q4_K_M | |
|---|---|---|
| Weights on disk | 29.98 GB | 17.74 GB |
VRAM (ollama ps, ctx 262,144) |
38 GB | 26 GB |
| Theoretical ceiling (864 GB/s ÷ weights) | 28.8 tok/s | 48.7 tok/s |
| Generation, short (21 tok) | 19.85 (19.9 / 19.8) | 25.68 (26.00 / 25.59 / 25.45) |
| Bandwidth efficiency (short) | 69% | 53% |
| Generation, long (31,073 tok) | 16.77 | 20.75 |
| Long-context slowdown | −15.5% | −19.2% |
| Prefill, long prompt | 533.6 tok/s (58.2 s) | 498.0 tok/s (62.4 s) |
| Total time (long) | 67.8 s | 69.5 s |
What I found
- 20 tok/s is not a fault. 100% GPU, no offload, normal temperatures. A 27B at Q8 reads ~30 GB per token; 864 GB/s caps that at 28.8 tok/s, and 69% efficiency is normal for llama.cpp on ROCm.
- My Q4 prediction missed: I expected ~35 and got 25.7. My mistake was assuming Q8’s 69% efficiency would carry
over. As the weights shrink, the work stops being bandwidth-bound and shifts to dequantization — unpacking the
weights — and K-quants (per-block scales and mins) are far more expensive to unpack than Q8_0’s
int8 × scale. - Efficiency drops with the quant. Same card, same model, same runtime: Q8_0 69%, Q4_K_M 53%. This was my first clean measurement that isolates the quantization alone.
- Prefill is actually slower on Q4_K_M. Prefill is compute-bound (matrix × matrix: the weights are read once and reused for thousands of tokens), so smaller weights don’t help, and the costlier unpacking is still there. Generation is bandwidth-bound (matrix × vector: every weight is reread for every token) — that’s where Q4 wins.
- Q4’s advantage erodes as the context grows. The q8_0 KV traffic is the same number of bytes for both models, so it takes a bigger share of Q4’s smaller per-token read. A cross-check with July: Qwen3.6-27B Q8_0 lost 14% at ~25K, Qwen3.8 loses 15.5% at 31K — the card and the stack behave consistently.
- The wait is prefill. 58–62 s before the first token at 31K, then 85 tokens in 5 s. No quant change fixes that; a shorter prompt or prefix caching does.
thinkchanges how many tokens you get, not how fast. Same question:think:false→ 572 tokens / 28.8 s,think:true→ 1,263 tokens / 64.0 s, both at ~19.8 tok/s.
A projection (not measured): 2× W7800 with BF16
This came up while I was planning hardware. llama.cpp/Ollama split multiple GPUs by layer, so the cards work one after another — bandwidths don’t add up, times do.
| Setup | Weights | Effective bandwidth | Theoretical | Expected |
|---|---|---|---|---|
| 1× W7800, Q8_0 | 30 GB | 864 GB/s | 28.8 | 19.9 (measured) |
| 1× W7800, Q4_K_M | 17.7 GB | 864 GB/s | 48.7 | 25.7 (measured) |
| 2× W7800, BF16, layer split | ~56 GB | 864 GB/s | 15.4 | ~10.5 |
| 2× W7800, BF16, tensor parallel (vLLM) | ~56 GB | 1,728 GB/s | 31 | ~20–25, if it runs on RDNA3 |
If I add a second card, it should be for capacity, not precision: for a 27B, BF16 vs Q8_0 is within noise on quality.