Orkun Soylu

2 min read

Dense 27B @168k token prompt

I fed a 168,084-token prompt to Qwen3.8-27B Q8_0 on my W7800: prefill took 11.7 minutes, and generation fell from 20.2 to 9.6 tok/s. It still recalled a marker from the very start perfectly.

Tested
Setup

Radeon Pro W7800 48 GB — bare-metal workstationBare metal, Linux, ROCm (Ollama) · AMD Ryzen 5 7600 (6C/12T, AVX-512) · 64 GB DDR5-6400

Runtime
Ollama 0.32.14, flash attention on, q8_0 KV cache, NUM_PARALLEL=1
Models

qwen3.8-27b-uncensoredQ8_0tested

Verdict

Long context works and stays accurate, but on this hardware it's a background job, not something you sit and wait for: the 'generation barely slows down' rule from 26K stops holding — at 168K speed halves.

Works

  • Marker placed at token 0 recalled correctly after 168K tokens — no silent truncation
  • 100% GPU throughout; VRAM stays at 39.5 GB because KV is pre-allocated
  • Prefill scales sub-linearly: 6.7× the context, 2.4× slower per token

Doesn’t

  • 702.9 s (11.7 min) to first token
  • Generation −52%: 20.2 → 9.64 tok/s
  • Card sits at its 230 W power limit for the whole prefill
Prompt
168,084tokens
Prefill time
11.7min
Generation at 168K
9.64tok/s
Marker recall
Correct

Generation speed vs filled context

higher is better

Dense 27B Q8_0 on the W7800; 26K and 31K points from earlier runs

Short prompt (~20 tok)
20.20 tok/sGeneration · 20.20 tok/s
~25K tokens (Qwen3.6, July)
16.80 tok/sGeneration · 16.80 tok/s
31K tokens (Qwen3.8, Aug 18)
16.77 tok/sGeneration · 16.77 tok/s
168K tokens
9.64 tok/sGeneration · 9.64 tok/s
Data
Generation
Short prompt (~20 tok)20.20 tok/s
~25K tokens (Qwen3.6, July)16.80 tok/s
31K tokens (Qwen3.8, Aug 18)16.77 tok/s
168K tokens9.64 tok/s

Prefill speed vs prompt length

higher is better
~25K tokens (Qwen3.6, July)
582.6 tok/sPrefill · 582.6 tok/s
31K tokens (Qwen3.8, Aug 18)
533.6 tok/sPrefill · 533.6 tok/s
168K tokens
239.1 tok/sPrefill · 239.1 tok/s
Data
Prefill
~25K tokens (Qwen3.6, July)582.6 tok/s
31K tokens (Qwen3.8, Aug 18)533.6 tok/s
168K tokens239.1 tok/s

Setup

  • Model: qwen3.8-27b-uncensored:q8_0, an abliterated Qwen3.8-27B with its f16 vision projector that I imported by hand from Hugging Face — architecturally identical to the official dense 27B.
  • Server: OLLAMA_CONTEXT_LENGTH=262144, KV_CACHE_TYPE=q8_0, FLASH_ATTENTION=1, NUM_PARALLEL=1 (I lowered it from 2 for this test), no num_ctx in the Modelfile.
  • Prompt: 780,112 characters of repeated filler with a marker string (MARKER-ALPHA-7391) at the very beginning and a question about it at the end. num_predict=40.
  • Where the numbers come from: Ollama’s own print_timing log lines, not the API response.

Results

Short prompt (~20 tok) Long prompt (168,084 tok)
Prefill — 239.1 tok/s (4.18 ms/token)
Prefill time instant 702.9 s (11.7 min)
Generation 20.2 tok/s 9.64 tok/s (103.7 ms/token)
Total instant 707.0 s
VRAM 39.5 / 48.3 GB same (KV pre-allocated)
Processor 100% GPU 100% GPU

The whole run, the card sat at its power limit: 229/230 W, 82 °C edge, fan 55%, 2,482 MHz.

What I found

  1. Generation halves at 168K. At 25–26K the slowdown was 14–16%, and I’d written it off as “not a big loss”. At 168K it’s 52%. The KV cache gets scanned end to end for every token, and that cost grows with the context. So my “prefill is the cost, generation barely changes” rule only holds within a range — true at 25K, false at 168K.
  2. Prefill slows down more gently. 582 tok/s at 25K → 239 tok/s at 168K: 6.7× the context, 2.4× slower per token. That’s sub-linear; attention’s quadratic term isn’t dominant yet.
  3. 168K means twelve minutes of waiting. Prefill (702.9 s) dwarfs generation (4.0 s). Using really long context interactively isn’t realistic on this hardware; batch and background jobs are. Anything that reuses a long cached prefix gains enormously.
  4. No accuracy loss. It recalled the marker from the start correctly after 168K tokens — nothing was silently cut from the window.
  5. One slot can go past 131,072. With NUM_PARALLEL=1, a single request processed 168,084 tokens. VRAM stayed exactly the same (39.5 GB) going from 2 slots to 1, which fits the idea that context is divided among slots — but I didn’t measure the 2-slot case.

Caveats

The filler text is repetitive (the log reports graphs reused = 126), so a real 168K document with varied text may be slower. num_predict=40 cut the answer off inside the thinking block, but by then the model had already read the marker correctly.