vLLM vs Ollama @W7800 (ROCm)
Of the seven quantization formats I tried, two ran on RDNA3 under vLLM 0.27. int8 W8A16 + MTP reached 42 tok/s; int4 W4A16 reached 57 tok/s but locked up the GPU at ~30K tokens. I removed vLLM — and Ollama with MTP matched it the next day.
- Tested
- Setup
Radeon Pro W7800 48 GB — bare-metal workstationBare metal, Linux, ROCm (Ollama) · AMD Ryzen 5 7600 (6C/12T, AVX-512) · 64 GB DDR5-6400
- Runtime
- vLLM 0.27 on ROCm 10 vs Ollama 0.34.0
- Models
Qwen3.8-27Bint8 W8A16 + MTPtested
Qwen3.8-27Bint4 W4A16tested
qwen3.8:27bQ8_0baseline
Verdict
Not worth it on RDNA3. Most formats have no ROCm kernels, the fast one hangs on long context, and with MTP you have to turn prefix caching off — so every agent turn re-reads the whole context. I removed it; Ollama stays.
Works
- int8 W8A16 + MTP: 42.3 tok/s and 850 tok/s prefill at 20K
- int4 W4A16: 56.6 tok/s with the full 262K context in VRAM
Doesn’t
- GGUF, 3-bit GSQ, Quark INT4, bitsandbytes and NVFP4: no ROCm kernels
- FP8 loads but has no hardware support on gfx1100: 0.8 tok/s
- int4: GPU kernel hung at ~30K tokens (100% busy, no progress for 12+ minutes)
- Prefix caching off with MTP: a 20K-token agent turn costs ~24 s every time
- Formats that ran
- 2 / 7
- Best usable decode (W8A16+MTP)
- 42.3tok/s
- FP8 decode (emulated)
- 0.8tok/s
Generation speed — same 4 prompts, streaming, temperature 1
higher is betterData
| Generation | |
|---|---|
| vLLM W8A16 + MTP | 42.3 tok/s |
| Ollama ISTA IQ3_S | 24.8 tok/s |
| Ollama Q8_0 (no MTP) | 19.8 tok/s |
20K-token prompt processing
higher is betterData
| Prefill | |
|---|---|
| vLLM W8A16 + MTP | 850 tok/s |
| Ollama Q8_0 | 531 tok/s |
| Ollama ISTA IQ3_S | 510 tok/s |
What runs (vLLM 0.27, ROCm 10, gfx1100)
I tried every format I could find for the model. Most didn’t make it:
| Format | Result |
|---|---|
| GGUF | ✗ the ROCm build lacks the ggml ops |
| FP8 (official Qwen) | ⚠ loads, but gfx1100 has no FP8 hardware → Triton emulation, 0.8 tok/s |
| 3-bit (ISTA GSQ) | ✗ no ROCm kernel for uint3b4 |
| Quark INT4 (AMD) | ✗ vLLM 0.27 has no Quark INT4 W4A16 scheme |
| bitsandbytes 4-bit, NVFP4 | ✗ |
| int8 W8A16 (with MTP) | ✓ 42 tok/s, but at 64K context; needs --dtype float16 and VLLM_DISABLED_KERNELS=ConchLinearKernel (Conch is 3× slower) |
| int4 W4A16 | ✓ 56.6 tok/s, 262K fits — but the GPU kernel locked up at ~30K tokens (py-spy in gpu_model_runner, GPU 100%, no progress for 12+ minutes) |
How it compares
The same 4 prompts, streaming, temperature 1:
| Generation | 20K-prompt processing | |
|---|---|---|
| vLLM W8A16 + MTP | 42.3 tok/s | 850 tok/s |
| Ollama Q8_0 | 19.8 tok/s | 531 tok/s |
| Ollama ISTA IQ3_S | 24.8 tok/s | 510 tok/s |
On raw generation speed, vLLM looked like a clear win.
Why I removed it anyway
- With MTP, prefix caching has to be off (an open upstream issue at the time), so every agent turn re-processes the whole context: 20K tokens ≈ 24 s, on every single turn. Ollama keeps the prompt cached within a session.
- The fast int4 path hangs on long context, and the stable int8 path stops at 64K.
- The very next day, Ollama’s own MTP support brought Q8_0 to the same 42.3 tok/s — with 262K context and caching. That took away vLLM’s only advantage.
So I deleted vLLM and its models; it’s Ollama only from here.