Orkun Soylu

3 min read

ISTA IQ3_S vs HauhauCS IQ3_XS @CVE triage

The first run of my tool-using CVE evaluation: both 3-bit 27B models scored 0.96. What separated them wasn't accuracy but language discipline — HauhauCS answered a third of my Turkish questions in English.

Tested
Setup

GeForce RTX 4070 Ti SUPER 16 GB — Proxmox VM (GPU passthrough)Proxmox VE guest with PCIe passthrough, Ubuntu 24.04, NVIDIA 595 open driver · Intel Core i5-13400 host; guest 8 vCPU · Guest 32 GB; host 64 GB DDR5

Runtime
Ollama 0.34.1 via my chat UI with tools; CVE MCP server 0.4.0
Models

Qwen3.8-27B GSQ-RCOISTA IQ3_Stested

Qwen3.8-27B AggressiveIQ3_XStested

Verdict

My question set is close to its ceiling — both models scored 0.96. The real differences were language drift (HauhauCS 15/45 answers in English, ISTA 0) and facts written from memory that no tool had returned (both).

Works

  • Zero invented CVE IDs from either model
  • Version-lookup tool called on every upgrade question (48/48)
  • ISTA: newest clean version recommended in 24/24 upgrade answers

Doesn’t

  • HauhauCS: 15 of 45 Turkish questions answered in English (3/3 on jinja2)
  • Both wrote wrong CVE details from memory that no tool had returned
  • First grader pass under-scored badly (0.73 / 0.65) — free-text grading needs hand calibration
Overall score
0.96 · 0.96
Non-Turkish answers
0 · 15/ 45
Newest clean upgrade
24 · 23/ 24
Seconds per answer
46 · 46

Answers in the wrong language

lower is better

Turkish question, non-Turkish answer, out of 45

  • ISTA IQ3_S
  • HauhauCS IQ3_XS
Wrong-language answers
0ISTA IQ3_S · 0
15HauhauCS IQ3_XS · 15
Data
ISTA IQ3_SHauhauCS IQ3_XS
Wrong-language answers015

Fix versions stated

Out of 42 fixed-version facts the answers had to give

  • ISTA IQ3_S
  • HauhauCS IQ3_XS
Correct
35ISTA IQ3_S · 35
37HauhauCS IQ3_XS · 37
Wrong
1ISTA IQ3_S · 1
0HauhauCS IQ3_XS · 0
Not stated
6ISTA IQ3_S · 6
5HauhauCS IQ3_XS · 5
Data
ISTA IQ3_SHauhauCS IQ3_XS
Correct3537
Wrong10
Not stated65

The evaluation

15 questions × 3 repetitions per model, asked in Turkish through my chat UI with tools enabled, served from the 4070 Ti SUPER (q8_0 KV, 65K context). The tools come from my CVE-triage MCP server (NVD, EPSS, CISA KEV, OSV, package versions, web search). I built the expected answers independently from NVD/OSV/deps.dev/KEV and froze them before the run; a script did the grading. 90 runs, zero errors or timeouts. The full write-up is in Methodology → CVE evaluation.

Results

ISTA GSQ-RCO IQ3_S HauhauCS Aggressive IQ3_XS
Overall 0.96 0.96
Upgrade: newest clean version recommended 24/24 23/24
Fix versions correct / wrong / not stated 35 / 1 / 6 37 / 0 / 5
KEV status correct 6/6 6/6
Prioritisation / non-existent CVE 3/3 / 3/3 3/3 / 3/3
Invented CVE IDs 0 0
Turkish question, non-Turkish answer 0/45 15/45
CVE written from memory (not in any tool result) 2 1
Tool calls / seconds per answer 2.6 / 46 s 2.8 / 46 s

The set is close to its ceiling: I can’t separate the two models on accuracy, and the upgrade questions are basically solved. Where they do differ is language consistency and knowledge from memory.

What I actually found (by reading the answers)

  1. Language drift — HauhauCS. 15 of 45 answers came back in English (3/3 on jinja2). ISTA never drifted. The first fix is a standard system prompt with a language rule.
  2. Wrong facts from memory — both models. The scores were high, yet details that weren’t in any tool output were wrong:
    • ISTA described CVE-2020-9488 as “JMSAppender RCE, CVSS 8.8”; it’s actually an SMTP-appender certificate validation issue, CVSS 3.7 (the JMSAppender RCE is CVE-2021-4104).
    • HauhauCS called a curl CVE (CVE-2023-27533) “a later MOVEit CVE”.
    • ISTA (log4j, run 1) gave CVE-2021-45046 the description and fix of CVE-2021-44832.
  3. A blind spot in my tools — Express. CVE-2022-24999 (through the qs dependency; NVD says express < 4.17.3) shows up in no answer, because OSV’s express record doesn’t include it. Vulnerabilities in transitive dependencies are a blind spot of the package scanner.
  4. A window bug in my tools — recent KEV entries. One of 13 KEV entries was missing from the tool output for both models in all 6 runs. Most likely the “last 7 days” boundary was computed in local time while the truth was in UTC. I fixed it in the next MCP release.
  5. NVD is wrong for MOVEit. NVD’s version bounds for MOVEit Transfer point at the next patch set; both models gave Progress’s official fixed versions, which are the correct ones. I recorded it as a documented vendor override.

Calibrating the grader

The first automatic pass gave ISTA 0.73 / HauhauCS 0.65. When I read every answer, most of that drop turned out to be my grader’s fault: a sentence-final period corrupted versions (“2.10.1.”), bold heading lines closed the recommendation section before the recommendation, “ransomware: unknown” was read as “not in KEV”, NVD’s inclusive bound (2.16.0 → 2.17.0) wasn’t understood, and only lines containing a CVE ID were looked at. After the fixes, the genuine errors above are still caught.

The upgrade metric checks whether the best version appears among the recommendations, not whether it’s the primary one. Free-text heuristics have to be re-checked by hand for every new answer style — which is exactly why I want a fixed, machine-readable summary block at the end of every answer.