Orkun Soylu

2 min read

ISTA IQ3_S vs Unsloth UD-IQ3_S @16 GB

On my RTX 4070 Ti SUPER under vLLM, ISTA's IQ3_S gets 1.2% lower perplexity from a smaller file and prefills 8% faster; Unsloth's decodes 5% faster and leaves a bit more room for KV.

Tested
Setup

GeForce RTX 4070 Ti SUPER 16 GB — Proxmox VM (GPU passthrough)Proxmox VE guest with PCIe passthrough, Ubuntu 24.04, NVIDIA 595 open driver · Intel Core i5-13400 host; guest 8 vCPU · Guest 32 GB; host 64 GB DDR5

Runtime
vLLM 0.29.0 + vllm-gguf-plugin (main d4c1f0d), torch 2.13 + CUDA 13.0
Models

Qwen3.8-27B GSQ-RCOISTA IQ3_S (3.50 bpw)tested

Qwen3.8-27B UDUnsloth UD-IQ3_S (3.52 bpw)tested

Verdict

ISTA GSQ-RCO IQ3_S: better perplexity from a smaller file, and faster prefill. Unsloth decodes slightly faster. Both fit in 16 GB — but only with tuned memory settings, and with ~20K tokens of KV in total.

Works

  • ISTA: wikitext-2 perplexity 6.543 vs 6.619, from a 0.25 GiB smaller file
  • ISTA: 833 vs 768 tok/s prefill on an 8K prompt
  • Both answer the sanity prompts correctly

Doesn’t

  • vLLM defaults do not start: only 0.3 GiB left for KV
  • Speed differences are single runs — indicative, not settled
  • Native 262K context is far out of reach: ~20K tokens of KV shared by all requests
Perplexity, ISTA vs Unsloth
6.543 / 6.619
Decode, 1 request
34.5 / 36.3tok/s
Prefill 8K
833 / 768tok/s
KV capacity
~20Ktokens

Wikitext-2 perplexity

lower is better

40 × 2,048-token chunks, 81,880 scored tokens, identical for both files

  • ISTA GSQ-RCO IQ3_S
  • Unsloth UD-IQ3_S
Perplexity
6.543ISTA GSQ-RCO IQ3_S · 6.543
6.619Unsloth UD-IQ3_S · 6.619
Data
ISTA GSQ-RCO IQ3_SUnsloth UD-IQ3_S
Perplexity6.5436.619

Throughput under vLLM

higher is better
  • ISTA GSQ-RCO IQ3_S
  • Unsloth UD-IQ3_S
Decode, 1 request
34.5 tok/sISTA GSQ-RCO IQ3_S · 34.5 tok/s
36.3 tok/sUnsloth UD-IQ3_S · 36.3 tok/s
Decode, 4 concurrent (total)
54.9 tok/sISTA GSQ-RCO IQ3_S · 54.9 tok/s
54.0 tok/sUnsloth UD-IQ3_S · 54.0 tok/s
Data
ISTA GSQ-RCO IQ3_SUnsloth UD-IQ3_S
Decode, 1 request34.5 tok/s36.3 tok/s
Decode, 4 concurrent (total)54.9 tok/s54.0 tok/s

Setup

  • vLLM 0.29.0, torch 2.13.0+cu130, NVIDIA driver 595.91.07. I had to build vllm-gguf-plugin from main (d4c1f0d082) — the PyPI release (0.0.5) didn’t support Qwen3.5/3.8.
  • Files, SHA-256 checked against Hugging Face LFS:
    • ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF → Qwen3.8-27B-GSQ-RCO-IQ3_S.gguf (10.96 GiB, 3.50 bpw)
    • unsloth/Qwen3.8-27B-GGUF → Qwen3.8-27B-UD-IQ3_S.gguf (11.21 GiB, 3.52 bpw, with an MTP block inside)
  • The same engine settings for both: gpu_memory_utilization=0.97, max_model_len=8192, max_num_seqs=4, max_num_batched_tokens=2048 (512 for perplexity), fp16 KV, CUDA graphs on, text only.
  • Perplexity: wikitext-2-raw test, the same 40 non-overlapping 2,048-token chunks for both, from vLLM’s prompt log-probs.

Results

ISTA GSQ-RCO IQ3_S Unsloth UD-IQ3_S
File 10.96 GiB 11.21 GiB
Weights in VRAM 11.99 GiB 11.86 GiB
KV capacity (serving settings) 19,757 tokens 20,720 tokens
Wikitext-2 perplexity (ctx 2048) 6.543 6.619 (+1.2%)
Decode, 1 request 34.5 tok/s 36.3 tok/s (+5%)
Decode, 4 concurrent (total) 54.9 tok/s 54.0 tok/s
Prefill, 8K-token prompt 833 tok/s 768 tok/s
Sanity prompts (17×23, Fibonacci, a Turkish fact) all correct all correct

Does a 27B fit in 16 GB?

Yes — with conditions. The card gives torch 15.57 GiB (the driver keeps ~430 MiB), and the weights take ~12 GiB.

Item vLLM default (0.92 / 8192) Tuned (0.97 / 2048)
Budget 14.33 GiB 15.10 GiB
Weights + non-torch ~12.1 GiB 12.12 GiB
Activation peak + CUDA graphs ~2.0 GiB 1.17 GiB
KV cache 0.3 GiB → engine refuses to start (an 8K request needs 0.81 GiB) 2.0 GiB → 19,757 tokens

The architecture is hybrid: only 16 of the 64 layers use classic attention (~64 KiB KV per token), which is why 2 GiB holds ~20K tokens. But that’s the total across all concurrent requests. Loading prints a stream of CUDACachingAllocator ... OOM warnings — those are just the allocator retrying, and the load succeeds.

How I read the comparison

  • ISTA’s 1.2% perplexity lead matches the direction and size of the gap in its own model card (7.07 vs 7.16 at a different context length). Perplexity isn’t task accuracy, though, and I didn’t re-run the card’s AIME25/GPQA/LiveCodeBench numbers.
  • Unsloth’s file is bigger on disk but smaller in VRAM: the MTP block inside it is skipped unless speculative decoding is on, which frees ~960 tokens of extra KV. ISTA ships a separate -mtp file instead.
  • The tensor mixes explain the speed split: ISTA spends fewer bits on the embedding (IQ2_S vs Q3_K) and the output layer (Q4_K vs Q5_K), and more on IQ4_XS layers in the body.