CVE system prompt v2: ISTA IQ3_S @0.98
A stricter grounding rule plus two tool fixes took ISTA to 0.98, with zero CVEs from memory and zero language drift. HauhauCS slipped to 0.92: it gives up researching upgrades too early.
- Tested
- Setup
GeForce RTX 4070 Ti SUPER 16 GB — Proxmox VM (GPU passthrough)Proxmox VE guest with PCIe passthrough, Ubuntu 24.04, NVIDIA 595 open driver · Intel Core i5-13400 host; guest 8 vCPU · Guest 32 GB; host 64 GB DDR5
- Runtime
- Ollama 0.34.1 via my chat UI with tools; CVE MCP server 0.5.2
- Models
Qwen3.8-27B GSQ-RCOISTA IQ3_Stested
Qwen3.8-27B AggressiveIQ3_XStested
Verdict
ISTA IQ3_S hits the ceiling of this set (0.98, no CVEs from memory, no drift), and it's now my default for security work on the 16 GB card. HauhauCS got better on language and on making things up, but it researches upgrades less thoroughly.
Works
- ISTA: 0.98, zero invented and zero memory CVEs, 40/42 fix versions correct
- KEV list question 1.00 for both once the window was cut per run (UTC)
Doesn’t
- HauhauCS: stopped before checking versions (Spring4Shell → 5.3.18, Jackson → 2.18.8)
- HauhauCS: one memory CVE despite the new rule, one fully English answer
- Overall, ISTA / HauhauCS
- 0.98 / 0.92
- Memory CVEs
- 0 / 1
- Fix versions correct
- 40 / 38of 42
- Seconds per answer
- 60 / 64
Overall score across three prompt iterations
higher is better- ISTA IQ3_S
- HauhauCS IQ3_XS
Data
| ISTA IQ3_S | HauhauCS IQ3_XS | |
|---|---|---|
| Baseline (no prompt) | 0.96 | 0.96 |
| Prompt v1 | 0.96 | 0.94 |
| Prompt v2 | 0.98 | 0.92 |
What I changed (vs v1)
- Prompt: I added “do not write a CVE ID that is not in a tool or web result”.
- MCP server 0.5.1: the version tool clamps
linesinstead of throwing an error. 0.5.2: the recent-KEV tool uses a UTC window and acceptssince. - The truth: I regenerated it right before the run (only the KEV list had changed), and graded the recent-KEV question against each run’s own UTC start date.
- 90 runs, zero errors or timeouts. I taught the grader the phrase “not listed” and re-graded three runs.
Results
| ISTA baseline → v1 → v2 | HauhauCS baseline → v1 → v2 | |
|---|---|---|
| Overall | 0.96 → 0.96 → 0.98 | 0.96 → 0.94 → 0.92 |
| Non-Turkish answers (real) | 0 → 0 → 0 | 15 → 1 → 1 |
| Invented CVE (not in NVD) | 0 → 1 → 0 | 0 → 1 → 0 |
| Memory CVE (real but untooled) | 2 → 2 → 0 | 1 → 0 → 1 |
| KEV list | 0.96 → 0.80* → 1.00 | 0.96 → 0.80* → 1.00 |
| Fix versions correct / wrong / missing | 35/1/6 → 39/0/3 → 40/0/2 | 37/0/5 → 40/0/2 → 38/1/3 |
| Upgrade: newest clean version | 24 → 24 → 23 | 23 → 22 → 22 |
| Version tool used | 24 → 24 → 24 | 24 → 22 → 22 |
| Tool errors | 0 → 1 → 0 | 0 → 4 → 2 |
| Seconds per answer | 46 → 57 → 60 | 46 → 62 → 64 |
* v1’s 0.80 was the truth drifting; with a per-run window both models get 6/6.
The remaining errors, read by hand
ISTA (all defensible): in Django run 3 it targeted 5.2.17 LTS rather than the newest 6.1.1 — a deliberate choice. Two answers were missing the summary block.
HauhauCS (real weaknesses):
- It stops researching upgrades too early. Spring4Shell run 3: no version lookup, “upgrade to 5.3.18” — that’s the first fix for this CVE, and it’s vulnerable to later ones. Jackson run 2: it recommended 2.18.8; the newest clean version is 2.22.2.
- A CVE from memory, again: MOVEit run 3 cites CVE-2023-34548 as “unverified”, despite the new rule.
- Language: xz run 3 came back entirely in English (the same question and the same repetition as in v1).
- Non-existent CVE, run 3: it correctly said “not in NVD”, but then passed along conflicting “CVSS 0.0” claims it found on the web.
Both: neither mentioned CVE-2022-24999 for Express (via the transitive qs) in any of the 6/6 runs — a blind spot
in my tools.
Where this leaves me
ISTA IQ3_S is at the top of this set. The next meaningful step is a harder question set — transitive dependencies, old-branch constraints (“we have to stay on Java 8”), conflicts between NVD and the vendor — plus a dependency graph in the tool to cover the Express blind spot.