Strata 125B MoE @48 GB vs Qwen3.8-27B
A purpose-built engine runs a 125B mixture-of-experts on my single W7800 by keeping the hot experts in VRAM and the rest in RAM. Tuned, the 3-bit build matches or beats my 27B baseline on quality and is 40–60% faster per token — but tool-heavy answers arrive 38% later.
- Tested
- Setup
Radeon Pro W7800 48 GB — Proxmox VM (GPU passthrough)Proxmox VE guest with PCIe passthrough, Ubuntu 26.04 · Intel Core i5-13400 host (6P+4E); guest 8 vCPU, 16 vCPU for the Strata test · Guest 40 GB (52 GB for the Strata test); host 64 GB DDR5-4800
- Runtime
- Strata 0.1.41 (commit fb58e0d), HIP with ROCm TheRock wheels; baseline Ollama 0.40.2
- Models
Qwen3.8-Flash-NextISTA GSQ-RCO IQ3_Stested
Qwen3.8-Flash-NextUnsloth UD-IQ4_XStested
qwen3.8:27b-q8_0-mtpQ8_0 + MTPbaseline
Verdict
IQ3_S with reasoning_effort 'medium' is the best Strata setup: CVE 0.975 vs 0.969 for the 27B, hardbench 20/20, 72 tok/s. UD-IQ4_XS is 10–23% slower and no better. Two settings are non-negotiable: medium effort (the default thinks forever) and an explicit sampling block (otherwise it decodes greedily).
Works
- 72 tok/s decode, 961 tok/s prefill at 31K — on one card plus system RAM
- CVE eval 0.975 (hand-corrected), 24/24 newest clean upgrades
- Thinkbench answers 2.4× sooner, hardbench 35% sooner than the 27B
- Zero SSD reads during generation; 95–99% of expert hits served from VRAM
Doesn’t
- Default reasoning effort is 'xhigh': 3 of 5 everyday prompts hit a 4K cap without answering
- Tool-heavy CVE answers take 84 s vs 61 s (2× the tool calls, longer answers)
- Very young project, one model family only, no GPU vision on AMD
- Cannot share the card with Ollama
- Decode (IQ3_S, tuned)
- 72.0tok/s
- Prefill 31K
- 961tok/s
- CVE score vs 27B
- 0.975 / 0.969
- Experts in VRAM
- 75%
Generation speed vs the 27B baseline
higher is betterThinking on, temperature 1; Strata at reasoning_effort medium
- Strata IQ3_S
- Strata UD-IQ4_XS
- Qwen3.8-27B Q8_0 + MTP
Data
| Strata IQ3_S | Strata UD-IQ4_XS | Qwen3.8-27B Q8_0 + MTP | |
|---|---|---|---|
| Thinkbench | 71.9 tok/s | 67.4 tok/s | 43.1 tok/s |
| Hardbench | 83.8 tok/s | 83.3 tok/s | 52.7 tok/s |
Seconds per answer
lower is better- Strata IQ3_S
- Strata UD-IQ4_XS
- Qwen3.8-27B Q8_0 + MTP
Data
| Strata IQ3_S | Strata UD-IQ4_XS | Qwen3.8-27B Q8_0 + MTP | |
|---|---|---|---|
| Thinkbench (everyday prompts) | 22.1 s | 27.3 s | 52.4 s |
| CVE evaluation (tools) | 83.8 s | 107.7 s | 60.8 s |
CVE evaluation score
higher is betterHand-corrected for grader false positives; same truth snapshot for all three
Data
| Score | |
|---|---|
| Strata IQ3_S | 0.975 |
| Qwen3.8-27B Q8_0 + MTP | 0.969 |
| Strata UD-IQ4_XS | 0.961 |
Tuning sweep (IQ3_S)
higher is betterDecode, 4 prompts × 1,024 tokens, thinking off, temperature 0
Data
| Decode | |
|---|---|
| Setup default (15 workers, 128K) | 57.2 tok/s |
| + async expert swapping | 58.6 tok/s |
| Pool workers 7 (instead of 15) | 72.7 tok/s |
| 262K context | 71.5 tok/s |
| Tuned: async + 7 workers + 262K + prefill 16K | 72.0 tok/s |
What Strata is
Strata (MIT; I tested commit fb58e0d, engine 0.1.41) is a C++/HIP engine built
to run exactly one model — Qwen3.8-Flash-Next: 125B parameters, 48 layers × 512 experts = 24,576 experts, 10
active per token — on a single GPU plus system RAM. The experts that get used a lot live in VRAM, the rest in RAM; a
29 GB per-layer-embedding table stays on the SSD; and a built-in MTP layer drafts tokens for speculative decoding. It
serves the OpenAI, Anthropic and Responses APIs, plus a web UI.
Where the project stood when I tested it: created 2026-09-24, ~19K stars, ~1.7K forks, ~570 open issues, ~200 commits in the three days before. It moves very fast — the latest commit at the time described a default that corrupted IQ3_S output after long prompts (caught by its own release test). I didn’t audit the code; outbound connections go to Hugging Face and its mirrors, GitHub, and AMD’s ROCm nightly index.
Baseline: qwen3.8:27b-q8_0-mtp (Ollama, 262K, 100% GPU), measured the same day with the same scripts in the
Coder390 test.
Setup
I temporarily gave the GPU VM 16 vCPU and 52 GB of RAM, plus a 300 GB model disk, and stopped Ollama so it couldn’t grab the card.
./setup.sh --check
./setup.sh --yes --family qwen --model IQ3_S --backend hip --no-start
--checkidentified the W7800 as gfx1100 and estimated that IQ3_S “fits in low-RAM mode, GPU holds ~79% of experts”.- No system ROCm: AMD’s TheRock wheels went into the venv, and the engine compiled locally in a few minutes.
- Download: IQ3_S, 2 shards, 83.6 GB; the MTP source (BF16, 5.2 GB) packed down to Q2_0. Plus a one-time 50 GB
experts.binfor low-RAM mode. - UD-IQ4_XS: I downloaded it at a pinned revision (3 shards, 93.7 GB, every SHA-256 matched) and packed it with
--compat-bf16(195 tensors rounded to BF16, max error 0.0144).
Where the experts live
In resident low-RAM mode, RAM holds only the experts the GPU doesn’t (it complements VRAM, it doesn’t copy it), and adaptive swapping moves the hot experts between RAM and VRAM as a conversation goes on. Nothing is read from the SSD during generation.
| 128K (setup default) | 262K (tuned) | |
|---|---|---|
| Experts in VRAM | 19,408 / 24,576 (79%), 36.98 GiB | 18,414 (75%), 35.07 GiB |
| Experts in RAM | 13.27 GiB | 18.18 GiB |
| VRAM free | 592 MiB | 590 MiB |
| GPU hit rate during decode | 95.1–98.7% | — |
Tuning sweep (IQ3_S)
Every row is a fresh engine restart. Decode = 4 prompts × 1,024 tokens (a Turkish story, Python code, Turkish technical text, English maths), temperature 0, thinking off. Prefill = 1K / 7.5K / 31K-token prompts.
| Trial | Change | Decode | Prefill 31K | Kept? |
|---|---|---|---|---|
| base | setup default (15 workers, 128K) | 57.2 | 859 | |
| kv-resident | resident KV streaming | — | 877 | ✗ decode after long prompts 61 → 42 |
| pin | pinned resident memory | — | 867 | ✗ no gain |
| async | --adapt-async 1 |
58.6 | 856 | ✓ +2.4% |
| spec 6 / 3 | draft length 6 / 3 | 55.7 / 56.7 | ✗ 4 is best | |
| workers 11 / 9 / 7 / 5 | --pool-workers N |
72.9 / 72.4 / 72.7 / 72.6 | 884 | ✓ +24% |
| 262K | --max-context 262144 |
71.5 | 867 | ✓ −1.4%, equal context to the 27B |
| k8v4 | --kv k8v4 |
71.3 | 867 | ✗ no difference |
| prefill 16K | --prefill 16384 |
71.1 | 961 | ✓ +9% prefill |
| tuned | async + 7 workers + 262K + prefill 16K | 72.0 | 961 | chosen |
- Decode isn’t limited by the CPU or RAM: 5 to 11 workers give the same result, and 15 workers collided with the host’s own threads. What speed really follows is MTP acceptance — 80–85% on code gives 86–89 tok/s, ~50% on Turkish prose gives ~55.
- Prefill is ~90% GPU compute. Experts that aren’t in VRAM get streamed over PCIe for every chunk, and bigger chunks spread that cost out.
- The card ran at full PCIe 4.0 x16. RAM speed (4800 vs 6400) would change these results by single-digit percent at most.
Quality — the reasoning-effort trap
I adapted the 27B’s thinkbench/hardbench scripts to the OpenAI API, with identical prompts, sampling (temperature 1, top_p 0.95, top_k 20, min_p 0) and caps.
Strata’s default reasoning effort is xhigh. The model’s template knows low, medium and xhigh (high maps
to xhigh), so any request that doesn’t send reasoning_effort gets the longest possible thinking:
| Thinkbench (4,096 cap) | IQ3_S default (xhigh) |
IQ3_S medium |
27B Q8 |
|---|---|---|---|
| Tokens per answer | 2,811 — 3 of 5 prompts hit the cap | 1,548 | 2,185 |
| Thinking share | 91% | 28% | 35% |
| Seconds per answer | 45.5 | 22.1 | 52.4 |
| Speed | 63.9 tok/s | 71.9 tok/s | 43.1 tok/s |
With a 16K cap, xhigh averaged 9,309 tokens (91% thinking) and 147 s per answer; on one prompt (a bash rsync backup
script) all three runs thought for 16K tokens and never got to an answer.
| Hardbench (32K cap) | IQ3_S xhigh |
IQ3_S medium |
UD-IQ4_XS medium |
27B Q8 |
|---|---|---|---|---|
| Correct | 20/20 | 20/20 | 20/20 | 19/20 |
| Tokens median / max | 1,516 / 28,899 | 1,843 / 5,289 | 1,476 / 5,049 | 1,446 / 4,996 |
| Total time | 1,122 s | 513 s | 472 s | 791 s |
| Speed | 67.5 | 83.8 | 83.3 | 52.7 tok/s |
medium got rid of the long tail without costing any accuracy. Both Strata builds sit at the ceiling of these two
tests, so I can’t separate them here — that’s what the CVE evaluation is for.
UD-IQ4_XS: four bits, no gain
Unsloth’s ~4-bit build carries 59.5 GB of experts (11% more than IQ3_S). With the same tuned settings and a 26 GiB RAM budget, every expert the GPU doesn’t hold fits in page-locked RAM — no file reads at all.
| IQ3_S | UD-IQ4_XS | |
|---|---|---|
| Experts in VRAM | 18,414 (75%), 35.1 GiB | 15,180 (62%), 34.3 GiB |
| Experts in RAM | 18.2 GiB | 21.2 GiB |
| Decode (4 prompts) | 72.0 tok/s | 64.7 tok/s (−10%) |
| Prefill 7.5K / 31K | 840 / 961 tok/s | 648 / 813 tok/s (−23% / −15%) |
Real work: the CVE evaluation
Strata served my chat UI over its OpenAI-compatible API, with two settings that turned out to be essential:
reasoning_effort: mediumas a server-side default — my UI doesn’t send an effort level.- An explicit
samplingblock (temperature 1, top_p 0.95, top_k 20, min_p 0). Without it, requests that don’t carry sampling parameters are decoded greedily.
15 questions × 3 runs per model, the same frozen truth snapshot and NVD cache as the 27B’s run, MCP server 0.5.2 with the same settings fingerprint, and the identical system prompt. I graded all 135 runs together.
| 27B Q8 + MTP | IQ3_S | UD-IQ4_XS | |
|---|---|---|---|
| Overall (grader) | 0.969 | 0.897 | 0.906 |
| Overall (hand-corrected) | 0.969 | 0.975 | 0.961 |
| Upgrade: newest clean | 23/24 | 24/24 | 23/24 |
| Fix versions correct / wrong / missing | 39/0/3 | 39/0/3 | 40/0/2 |
| KEV / KEV list / prioritisation | 6/6 · 1.00 · 3/3 | 6/6* · 1.00 · 3/3 | 6/6* · 1.00 · 3/3 |
| Non-existent CVE handled | 3/3 | 3/3* | 2/3* |
| Invented / memory CVEs | 0 / 0 | 0 / 0 | 2** / 0 |
| Empty answers | 0 | 0 | 1 |
| Non-Turkish answers | 0/45 | 0/45 | 0/45 |
| Tool calls per answer | 3.8 | 6.4 | 7.3 |
| Seconds per answer | 60.8 | 83.8 | 107.7 |
* corrected by hand (grader errors, below). ** real but minor.
Where I corrected the grader. On the non-existent CVE (CVE-2025-99999), every Strata answer said it doesn’t exist and flagged a fake web repository as fabricated — but then also helpfully offered “you may mean CVE-2025-9999”, with that real record’s CVSS. My grader’s “did you mean” exception works line by line, so the CVSS line got scored as a claim about the fake CVE. Similarly, a KEV answer that correctly said “yes, listed” was marked wrong because a separate line about a different, older CVE contained a negation. What I left as scored: one UD-IQ4_XS answer suggested two IDs that don’t exist as alternatives (0.5), and one ended with no answer after six tool calls (0.125).
How it behaves. Strata’s model researches harder — 6.4 tool calls on average (up to 13) and long, sourced final answers (4–7K tokens). It also runs into my UI’s tool-round limit, where the UI removes the tool list for one last turn. Because the tool definitions sit at the start of the prompt, that change invalidates the engine’s prefix cache, and ~30K tokens get re-read (~30 s per question). Between normal tool rounds the prefix cache works fine and only the new tool result (1–9K tokens) is processed.
My verdict
- The best Strata configuration: IQ3_S +
reasoning_effort: medium. UD-IQ4_XS is 10–23% slower on this hardware and showed no quality advantage anywhere. - Against the 27B Q8: quality equal or slightly ahead (CVE 0.975 vs 0.969, hardbench 20/20 vs 19/20, 24/24 upgrade advice), +40–60% tokens per second, and answers 2.4× sooner on everyday prompts and 35% sooner on hardbench. But tool-heavy real work arrives 38% later — twice the tool calls, longer answers, and the cache miss on the final tool round.
- Caveats: set the effort and sampling on the server; the project is very young; it runs one model family only; vision on AMD is CPU-only (I didn’t test it); and it can’t share the GPU with Ollama.
What came next — the same engine split across two GPUs: Strata on W7800 + RX 7900 XTX.