ISTA IQ3_S: general capability
On my 16 GB card, ISTA's 3-bit Qwen3.8-27B nearly maxes out everyday benchmarks: 40/41 HumanEval, 49/50 GSM8K, 9/9 long-context recall, 15/15 tool calls. The one weakness I could see is Turkish reading comprehension (36/40 vs 40/40 in English).
- Tested
- Setup
GeForce RTX 4070 Ti SUPER 16 GB — Proxmox VM (GPU passthrough)Proxmox VE guest with PCIe passthrough, Ubuntu 24.04, NVIDIA 595 open driver · Intel Core i5-13400 host; guest 8 vCPU · Guest 32 GB; host 64 GB DDR5
- Runtime
- Ollama 0.34.1, 65K context, q8_0 KV cache, model's default sampling
- Models
Qwen3.8-27B GSQ-RCOISTA IQ3_Stested
Verdict
3-bit compression costs almost nothing on ordinary work — code, maths, long-context retrieval, vision and tool calling are all near-perfect. The set is too easy to rank models with; the only signal is a 10-point gap between English and Turkish reading.
Works
- HumanEval 40/41, GSM8K 49/50 (the miss is an ambiguous question)
- Needle recall 9/9 at 16K / 32K / 61K
- Tool calling 15/15 including 4 cases where it must not call
Doesn’t
- Belebele Turkish 36/40 vs English 40/40 on the same questions
- Counted 8 circles instead of 9 with thinking off
- HumanEval (41 problems)
- 98%
- GSM8K (50)
- 98%
- Tool calling (15 scenarios)
- 15/15
- Decode, short / 61K
- 43.5 / 33tok/s
Accuracy by area
higher is betterData
| Accuracy | |
|---|---|
| Code — HumanEval (41) | 97.6% |
| Maths — GSM8K (50) | 98.0% |
| Reading — Belebele English (40) | 100.0% |
| Reading — Belebele Turkish (40) | 90.0% |
| Long-context needles (9) | 100.0% |
| Vision — generated images (12) | 91.7% |
| Tool calling (15) | 100.0% |
Setup
I tested straight against Ollama’s /api/chat (no chat UI in between), with 65K context, q8_0 KV and the model’s
default sampling (temperature 1, top_k 20, top_p 0.95). Every answer was scored mechanically, and I pulled the samples
from the Hugging Face datasets server.
Results
| Area | Test | Result | Thinking | Per answer |
|---|---|---|---|---|
| Code | HumanEval, 41 problems (every 4th), unit tests in a network-less container | 40/41 (98%) | on | 19.8 s |
| Maths | GSM8K, 50 questions | 49/50 (98%) — the miss hinges on “three times more” | on | 7.3 s |
| Reading, English | Belebele eng_Latn, 40 questions |
40/40 | off | 0.3 s |
| Reading, Turkish | Belebele tur_Latn, the same 40 questions |
36/40 (90%) | off | 0.4 s |
| Long context | 3 facts at 10% / 50% / 90% depth; 16K / 32K / 61K tokens | 9/9 | off | 12 / 24 / 54 s |
| Vision | Generated images: text ×4, counting ×4, bar chart ×4 | 11/12 — said 8 circles for 9 | off | 0.6 s |
| Tool calling | 15 scenarios: Turkish/English, argument normalisation (IATA, ISO dates, “19:00”), parallel calls, 4 must-not-call cases | 15/15 | on | 2.9 s |
Speed: decode ~43.5 tok/s at short context, 33 tok/s at 61K. Prefill ~1,520 tok/s at 16K, ~1,190 tok/s at 61K.
What I make of it
For everyday use this set is maxed out: despite the 3-bit compression, code, maths, long context and tool calling are almost error-free. The one consistent weakness is Turkish reading comprehension — on the very same questions, English scored 40/40 and Turkish 36/40, and none of the four Turkish misses is wrong in English. A fiddly visual task (counting) slipped once with thinking off.
To actually rank models I need harder sets (LiveCodeBench, MATH-500, multi-step agent tasks) — or real work. That’s why most of my later comparisons on this site use the CVE evaluation.