Orkun Soylu

2 min read

Max context in 48 GB: five models

With one model loaded at a time, both Qwen3.6 models get their full 262K context entirely in VRAM on my W7800. Gemma 4 31B doesn't — above 64K it spills to the CPU.

Tested
Setup

Radeon Pro W7800 48 GB — bare-metal workstationBare metal, Linux, ROCm (Ollama) · AMD Ryzen 5 7600 (6C/12T, AVX-512) · 64 GB DDR5-6400

Runtime
Ollama 0.31.1 (ROCm), flash attention on, q8_0 KV cache
Models

qwen3.6:27bQ8_0tested

qwen3.6:35b-a3bQ8_0tested

gemma4:31b-itQ8_0tested

translategemma:27b-itQ8_0tested

Verdict

Run a 48 GB card one big model at a time: the Qwen models keep their whole 262K window on the GPU with 7–10 GB to spare. Gemma 4 31B needs a 64K cap to stay fully in VRAM.

Works

  • qwen3.6:27b Q8_0 at 262K: 38.66 GB, fully on GPU
  • qwen3.6:35b-a3b Q8_0 at 262K: 41.29 GB, fully on GPU
  • A global 262K default does not break models with smaller windows — Ollama silently clamps them

Doesn’t

  • The two Qwen models never fit together: their weights alone are 65.9 GB
  • Gemma 4 31B at 262K offloads 12.9 GB to the CPU and slows down
Card VRAM (rocm-smi)
48.3GB
27B KV cache per 1K tokens
~40MB
35B-A3B KV cache per 1K tokens
~13.6MB
Gemma 4 offload-free limit
65,536tokens

VRAM at maximum context

lower is better

Measured by actually loading each model, not by formula. Card total 48.3 GB.

  • VRAM
  • CPU offload
qwen3.6:27b Q8_0 @ 262K
38.66 GBVRAM · 38.66 GB
0.00 GBCPU offload · 0.00 GB
qwen3.6:35b-a3b Q8_0 @ 262K
41.29 GBVRAM · 41.29 GB
0.00 GBCPU offload · 0.00 GB
translategemma:27b Q8_0 @ 131K
31.85 GBVRAM · 31.85 GB
0.00 GBCPU offload · 0.00 GB
gemma4:31b Q8_0 @ 262K
30.88 GBVRAM · 30.88 GB
12.87 GBCPU offload · 12.87 GB
Data
VRAMCPU offload
qwen3.6:27b Q8_0 @ 262K38.66 GB0.00 GB
qwen3.6:35b-a3b Q8_0 @ 262K41.29 GB0.00 GB
translategemma:27b Q8_0 @ 131K31.85 GB0.00 GB
gemma4:31b Q8_0 @ 262K30.88 GB12.87 GB

Gemma 4 31B: where offload starts

Total model size vs the part that lands in VRAM

  • Total size
  • In VRAM
262,144 tokens
43.75 GBTotal size · 43.75 GB
30.88 GBIn VRAM · 30.88 GB
131,072 tokens
37.88 GBTotal size · 37.88 GB
35.42 GBIn VRAM · 35.42 GB
65,536 tokens
35.34 GBTotal size · 35.34 GB
35.34 GBIn VRAM · 35.34 GB
32,768 tokens
34.77 GBTotal size · 34.77 GB
34.77 GBIn VRAM · 34.77 GB
16,384 tokens
34.48 GBTotal size · 34.48 GB
34.48 GBIn VRAM · 34.48 GB
Data
Total sizeIn VRAM
262,144 tokens43.75 GB30.88 GB
131,072 tokens37.88 GB35.42 GB
65,536 tokens35.34 GB35.34 GB
32,768 tokens34.77 GB34.77 GB
16,384 tokens34.48 GB34.48 GB

The question

If the card holds one model at a time, how far can I push each model’s context before anything spills to the CPU? And does raising the global context default break the smaller models?

Why two models never fit together

I loaded each model at two different context sizes and fitted a straight line through the results. That gives the KV-cache growth per token and the weight footprint at zero context:

Model Weights (context → 0) KV cache per 1K tokens
qwen3.6:27b-q8_0 28.27 GB ~40 MB
qwen3.6:35b-a3b-q8_0 (MoE) 37.63 GB ~13.6 MB

Together the weights alone are 65.9 GB — 17.6 GB more than the 48.3 GB card, before a single token of context. OLLAMA_MAX_LOADED_MODELS=2 changes nothing: Ollama just evicts one of them.

The MoE’s KV cache is a third of the dense model’s per token. That’s why its full window costs only a little more than the dense model’s, even though its weights are bigger.

One model at a time: every model at its maximum

Model Architecture max VRAM at max (measured) Headroom
qwen3.6:27b-q8_0 262,144 38.66 GB, no offload ~9.6 GB
qwen3.6:35b-a3b-q8_0 262,144 41.29 GB, no offload ~5.5–7 GB
gemma4:31b-it-q8_0 262,144 30.88 GB VRAM + 12.9 GB CPU spills — slower
translategemma:27b-it-q8_0 131,072 (clamped) 31.85 GB, no offload comfortable

Setting OLLAMA_CONTEXT_LENGTH=262144 globally turned out to be safe: when a model’s own maximum is smaller, Ollama quietly clamps the request instead of failing. Gemma 4 is the exception only because its architectural maximum is 262K.

Gemma 4: stepping down to find the limit

num_ctx Total In VRAM CPU offload
262,144 43.75 GB 30.88 GB 12.87 GB
131,072 37.88 GB 35.42 GB 2.46 GB
65,536 35.34 GB 35.34 GB 0
32,768 34.77 GB 34.77 GB 0
16,384 34.48 GB 34.48 GB 0

So on this card, the practical offload-free ceiling for Gemma 4 31B Q8_0 is 65,536.

Overriding the default per model

The global setting is only a default. There are two ways to pin a model lower:

  1. Per request — "options": {"num_ctx": 65536} in the API body. I checked that it overrides the 262K default.

  2. As a new tag — a two-line Modelfile:

    FROM gemma4:31b-it-q8_0
PARAMETER num_ctx 65536

    ollama create gemma4:31b-it-q8_0-64k -f Modelfile loads at 65,536 every time. I measured size == size_vram (35.34 GB): zero CPU offload.

What I didn’t measure

I didn’t load medgemma:27b (same architecture as TranslateGemma) or qwen3-embedding at their maximums. I’d expect the same behaviour, but I haven’t verified it.