Own Q6_K (Turkish imatrix) vs Q8_0 vs ISTA IQ3_S
I made a Q6_K with a 145K-token calibration set (40% Turkish), and it stays 10× closer to Q8_0 than ISTA's 3-bit does (KLD 0.0056 vs 0.056). My targeted imatrix beat a bigger generic one by 4–7%. But with MTP on, Q6_K is slower than Q8_0.
- Tested
- Setup
Radeon Pro W7800 48 GB — bare-metal workstationBare metal, Linux, ROCm (Ollama) · AMD Ryzen 5 7600 (6C/12T, AVX-512) · 64 GB DDR5-6400
- Runtime
- llama.cpp b11081 (ROCm build) for quantization; Ollama 0.34.0 for speed
- Models
Qwen3.8-27BQ6_K + Turkish imatrixtested
qwen3.8:27bQ8_0baseline
Qwen3.8-27B GSQ-RCOISTA IQ3_Sreference
Verdict
Making your own quant is cheap (minutes), and a targeted imatrix measurably helps — but on this card the Q6_K wasn't worth deploying: MTP verification is compute-heavy, and unpacking Q6_K made it slower than Q8_0 + MTP.
Works
- Q6_K KL divergence to Q8_0: 0.0056 (95.7% top-1 agreement)
- My 284-chunk Turkish imatrix beats ISTA's published 1,000-chunk one: KLD 0.0185 vs 0.0200
- Whole pipeline in ~11 minutes on one GPU
Doesn’t
- Q6_K + MTP slower than Q8_0 + MTP (Turkish poem 22.0 vs 26.5, code 36.2 vs 46.8 tok/s)
- Q8_0 ignores the imatrix entirely — it only matters at Q6_K and below
- KLD to Q8_0, Q6_K-TR
- 0.0056
- KLD to Q8_0, ISTA IQ3_S
- 0.056
- imatrix gain, targeted vs generic
- 4–7%
- Imatrix compute
- 7min
KL divergence to Q8_0 (Q4_K_M, two imatrices)
lower is better- My Turkish imatrix (284 chunks)
- ISTA's published imatrix (1,000 chunks)
Data
| My Turkish imatrix (284 chunks) | ISTA's published imatrix (1,000 chunks) | |
|---|---|---|
| Mixed evaluation set | 0.0185 | 0.0200 |
| Turkish-only set | 0.0163 | 0.0171 |
With MTP on: Q6_K vs Q8_0
higher is better- Q8_0 + MTP
- Q6_K + MTP
Data
| Q8_0 + MTP | Q6_K + MTP | |
|---|---|---|
| Turkish poem | 26.5 tok/s | 22.0 tok/s |
| Code | 46.8 tok/s | 36.2 tok/s |
How I made it
- Source:
unsloth/Qwen3.8-27B-GGUFBF16 (51 GB, hash checked). - Tools: a llama.cpp b11081 ROCm build. There’s no system ROCm on that box, so I borrowed the missing libraries from the runtime that ships with Ollama.
- Calibration set (145K tokens): 40% Turkish (Wikipedia + FineWeb-2), 25% a widely used community calibration set, 20% code from my own repositories, 15% my own documentation.
- How long it took: Q8 reference 47 s · imatrix 7 min (GPU) · Q6_K 2 min · Ollama import 50 s.
How close it stays to Q8_0
| Model | KLD to Q8_0 | Generation | Note |
|---|---|---|---|
| Q6_K + Turkish imatrix | 0.0056 (top-1 95.7%) | 22.7 tok/s | |
| ISTA IQ3_S | 0.056 | — | 10× further from Q8_0 |
| Q8_0 (reference) | 0 | 19.8 tok/s |
Targeted vs generic importance matrix (both at Q4_K_M)
| Evaluation set | My TR imatrix (284 chunks) | ISTA’s published (1,000 chunks) |
|---|---|---|
| Mixed | 0.0185 | 0.0200 |
| Turkish only | 0.0163 | 0.0171 |
Less data, but targeted, beat more generic data by 4–7%.
Q8_0 ignores the imatrix. llama.cpp’s
quantize_q8_0explicitly throws the importance weights away. An imatrix only matters at Q6_K and below.
The MTP twist
Without speculative decoding, Q6_K is faster than Q8_0 (22.7 vs 19.8 tok/s), just as you’d expect from smaller weights. With MTP it flips: Turkish poem 22.0 vs 26.5, code 36.2 vs 46.8 tok/s. MTP verification processes ~5 tokens at once, which shifts the work from bandwidth to compute — and there, unpacking K-quants dominates. It’s the same effect I saw in Q8_0 vs Q4_K_M prefill.
I deleted the Q6_K after measuring it; Q8_0 + MTP stayed the default.