Orkun Soylu

3 min read

vLLM vs Ollama @RTX 4070 Ti SUPER

For a single user, Ollama beat vLLM on every axis I measured: 28% faster decode, 1.9× faster prefill, 2.5× the context in VRAM, a 3-second start and less energy per token.

Tested
Setup

GeForce RTX 4070 Ti SUPER 16 GB — Proxmox VM (GPU passthrough)Proxmox VE guest with PCIe passthrough, Ubuntu 24.04, NVIDIA 595 open driver · Intel Core i5-13400 host; guest 8 vCPU · Guest 32 GB; host 64 GB DDR5

Runtime
vLLM 0.29.0 (GGUF plugin) vs Ollama 0.34.1
Models

Qwen3.8-27B GSQ-RCOISTA IQ3_Stested

Verdict

Ollama. vLLM's real strength — lots of concurrent requests — is wasted on a single-user box I only switch on when I need it, and on this card it only runs a 27B through an experimental GGUF plugin built from source, with forced memory settings.

Works

  • Ollama decode 43.9 vs 34.4 tok/s; prefill 1,578 vs 833 tok/s at 8K
  • Ollama keeps 49K tokens fully on the GPU vs ~20K for vLLM
  • Ollama: 2.8 s to start vs 64–199 s
  • ~8.3 vs ~9.7 J/token at the wall

Doesn’t

  • My first Ollama reading (26.4 tok/s, one request after load) was wrong — repeated runs gave 43.9
  • Wall readings are meter ranges; CPU power under load was not measured
Decode, Ollama vs vLLM
43.9 / 34.4tok/s
Context fully in VRAM
49K / ~20Ktokens
Start time
2.8 s / 64–199 s
Energy per token
8.3 / 9.7J

Throughput, same GGUF file

higher is better
  • Ollama 0.34.1
  • vLLM 0.29.0
Decode, 1 request (8–9 × 1,024 tokens)
43.9 tok/sOllama 0.34.1 · 43.9 tok/s
34.4 tok/svLLM 0.29.0 · 34.4 tok/s
Prefill, 8K-token prompt
1,578.0 tok/sOllama 0.34.1 · 1,578.0 tok/s
833.0 tok/svLLM 0.29.0 · 833.0 tok/s
Data
Ollama 0.34.1vLLM 0.29.0
Decode, 1 request (8–9 × 1,024 tokens)43.9 tok/s34.4 tok/s
Prefill, 8K-token prompt1,578.0 tok/s833.0 tok/s

Power at the wall (whole host)

lower is better

Midpoint of the plug-in meter's range

Host on, all VMs off (no GPU driver, fans spinning)
53.5 WWall power · 53.5 W
VM on, GPU idle, no engine
42.5 WWall power · 42.5 W
Ollama, model loaded, idle
42.0 WWall power · 42.0 W
vLLM, model loaded, idle
46.5 WWall power · 46.5 W
vLLM generating
335.0 WWall power · 335.0 W
Ollama generating
365.0 WWall power · 365.0 W
Data
Wall power
Host on, all VMs off (no GPU driver, fans spinning)53.5 W
VM on, GPU idle, no engine42.5 W
Ollama, model loaded, idle42.0 W
vLLM, model loaded, idle46.5 W
vLLM generating335.0 W
Ollama generating365.0 W

Setup

I ran Ollama 0.34.1 from a tarball in my user directory (no systemd service). I imported the GGUF with RENDERER qwen3.8 + PARSER qwen3.5 and the official sampling parameters, copied from the registry’s qwen3.8:27b-q8_0. fp16 KV, flash attention on, num_ctx 20000. vLLM ran exactly as in the quant comparison. I didn’t measure perplexity or 4-way concurrency on Ollama (no prompt log-probs; I’m a single user anyway).

Results

vLLM 0.29.0 + GGUF plugin Ollama 0.34.1
Engine start 64 s (warm compile cache) – 199 s (cold) 2.8 s
Decode, 1 request (8–9 × 1,024 tokens) 34.4 tok/s 43.9 tok/s
Prefill, 8K tokens 833 tok/s 1,578 tok/s
Context fully in VRAM (fp16 KV) ~20K tokens 49K
VRAM 15.6 GiB, always (pre-allocates) 12.1 GiB at 20K, 13.9 GiB at 49K
4 concurrent requests (total) 54.9 tok/s not measured
Sanity prompts / thinking separated correct / — identical output / yes

Ollama doesn’t reserve any activation/graph headroom and sizes the KV cache to num_ctx — that’s where its extra context comes from.

Ollama context capacity on 16 GB

num_ctx 8,192 20,000 32,768 49,152 65,536 98,304
Model size 11.35 GiB 12.09 GiB 12.87 GiB 13.89 GiB 15.57 GiB 17.77 GiB
On GPU 100% 100% 100% 100% 91% 79%

What vLLM needed to run a GGUF

  • GGUF support lives in a separate plugin. The PyPI release didn’t know Qwen3.8, so I built it from GitHub with the CUDA toolkit installed — and now I have to track updates by hand.
  • The plugin reads its config from the tokenizer directory and refuses to load without the vision projector → mmproj + language_model_only.
  • The memory settings have to be forced, and PATH has to include the venv and CUDA so FlashInfer can JIT-compile on the first request. vLLM’s own docs call GGUF support “highly experimental”.

Ollama: download, ollama create (21 s), done.

A correction. My first comparison measured Ollama at 26.4 tok/s from a single request right after loading, and I concluded “vLLM is ~30% ahead”. For the power test I repeated the request nine times: every run gave ~44 tok/s, while vLLM stayed at 34.4–34.5 in both sessions. The right conclusion is Ollama ~28% ahead.

Power at the wall

A plug-in meter on the whole host (Intel i5-13400, 64 GB, 850 W Gold PSU) with only this VM running. The load: back-to-back single requests of 1,024 tokens each, thinking off, temperature 0.7, for 3.5 minutes.

State Wall GPU (nvidia-smi)
Host off (Wake-on-LAN standby) 1.6–2.0 W —
Host on, all VMs off 52–55 W no driver, D3hot, fans spinning
VM on, GPU idle, no engine 42–43 W P8, 13.3 W, fans stopped
Ollama, model loaded, idle 42 W P8, 13.7 W, fans stopped
vLLM, model loaded, idle 46–47 W P8, 14.8 W; CPU package 2.0 W (vs 0.5–0.6 W)
Ollama generating (43.9 tok/s) 360–370 W 282.5 W avg, fans ≤ 69%, ≤ 74 °C → ~8.3 J/token
vLLM generating (34.4 tok/s) 330–340 W 283.8 W avg, fans ≤ 79%, ≤ 74 °C → ~9.7 J/token
  • Shutting the VM down actually raises the draw by ~11 W. Without a driver, the card’s VBIOS keeps the fans spinning even in D3hot; once the guest driver loads, it drops to P8 within ~10 s and stops them.
  • Keeping a model in VRAM costs nothing — both engines let the GPU idle at P8.
  • vLLM idles ~4–5 W higher, because its engine keeps a CPU core half awake.
  • Under load the GPU draws the same (~283 W of a 285 W limit). Ollama’s extra ~30 W at the wall is most likely on the CPU side (I didn’t measure it), and it still uses less energy per token because it generates faster.