Orkun Soylu

3 min read

CVE system prompt v1

One shared system prompt took HauhauCS's English answers from 15 down to 1 and improved fix-version accuracy for both models. The facts from memory stayed — now labelled 'unverified', but still wrong.

Tested
Setup

GeForce RTX 4070 Ti SUPER 16 GB — Proxmox VM (GPU passthrough)Proxmox VE guest with PCIe passthrough, Ubuntu 24.04, NVIDIA 595 open driver · Intel Core i5-13400 host; guest 8 vCPU · Guest 32 GB; host 64 GB DDR5

Runtime
Ollama 0.34.1 via my chat UI with tools; CVE MCP server 0.5.0
Models

Qwen3.8-27B GSQ-RCOISTA IQ3_Stested

Qwen3.8-27B AggressiveIQ3_XStested

Verdict

A prompt fixes behaviour, not knowledge: answer language and the summary format were solved straight away and fix-version accuracy went up, but both models kept attaching real-but-unrelated CVEs from memory. 'Mark it unverified' is too weak a rule.

Works

  • HauhauCS wrong-language answers: 15 → 1
  • Fix versions correct: ISTA 35 → 39, HauhauCS 37 → 40 (of 42)
  • Machine-readable summary block in 45/45 answers for both

Doesn’t

  • +10–16 s per answer (summary block and extra verification calls)
  • Memory CVEs still appear, now flagged 'unverified' and sometimes copied into the summary
  • A time-sensitive question drifted because the truth was frozen 6 hours before the run
Overall, ISTA / HauhauCS
0.96 / 0.94
Wrong-language answers
0 / 1/ 45
Summary block present
45 / 45
Seconds per answer
57 / 62

Fix versions correct, baseline → prompt v1

higher is better

Out of 42

  • Baseline (no prompt)
  • Prompt v1
ISTA IQ3_S
35Baseline (no prompt) · 35
39Prompt v1 · 39
HauhauCS IQ3_XS
37Baseline (no prompt) · 37
40Prompt v1 · 40
Data
Baseline (no prompt)Prompt v1
ISTA IQ3_S3539
HauhauCS IQ3_XS3740

Seconds per answer

lower is better
  • Baseline (no prompt)
  • Prompt v1
ISTA IQ3_S
46 sBaseline (no prompt) · 46 s
57 sPrompt v1 · 57 s
HauhauCS IQ3_XS
46 sBaseline (no prompt) · 46 s
62 sPrompt v1 · 62 s
Data
Baseline (no prompt)Prompt v1
ISTA IQ3_S46 s57 s
HauhauCS IQ3_XS46 s62 s

What I changed

Just one thing: a shared system prompt for both models. Answer in the user’s language; take CVE IDs, versions and scores only from tool output, and mark anything else “unverified”; finish with a fixed ```summary block. Same frozen truth, same 15 questions × 3 runs. The MCP server moved from 0.4.0 to 0.5.0 (the tools behave the same; an NVD API key is now set). 90 runs, zero errors. I added summary-block checks to the grader, but they don’t count toward the score.

Results (baseline → prompt v1)

ISTA HauhauCS
Overall 0.96 → 0.96 0.96 → 0.94
Non-Turkish answers (real) 0 → 0 15 → 1
Summary block – → 45/45 – → 45/45
KEV field in the summary correct – → 6/6 – → 6/6
Fix versions correct / wrong / missing 35/1/6 → 39/0/3 37/0/5 → 40/0/2
Upgrade: newest clean version 24 → 24 23 → 22
Memory CVEs (unflagged → flagged) 2 unflagged → 3, all “unverified” 1 unflagged → 1, “unverified”
Tool calls / seconds per answer 2.6 / 46 → 2.9 / 57 2.8 / 46 → 3.5 / 62

The language grader counted two ISTA answers as English because of English words inside tables. I read both — they’re Turkish. The only real drift was a single HauhauCS answer.

Why some scores went down

  1. The recent-KEV question went 0.96 → 0.80 — but it was the truth that moved, not the models. I froze the truth at 18:37 UTC and ran the evaluation ~6 hours later. The “last 7 days” window slid in between: 4 old entries dropped out and 1 new one came in. Both models reproduced the tool output exactly in 6/6 runs. Lesson learned: time-dependent questions need their truth generated at run time.
  2. HauhauCS left Spring4Shell vulnerable in 2/3 runs. It skipped the version-lookup tool, said “latest 5.3.x”, and honestly marked the number as unverified. That’s not false — it’s incomplete research (there’s no clean 5.3.x in the open-source line; the best target is 7.0.9).
  3. Express 0.62 / 0.75 — the transitive-dependency blind spot I already knew about; no prompt can fix that.

What the prompt didn’t fix

  • The CVEs from memory are still there, now flagged — and some leak into the summary. ISTA attached CVE-2023-46805 (Ivanti) and CVE-2024-1709 (ScreenConnect) to MOVEit; HauhauCS named a CVE-2024-17362 that doesn’t exist. Every one of them was marked “unverified”, and every one was wrong. The rule has to be “never write a CVE ID that isn’t in a tool result”, not “flag it”.
  • The summary’s target vs the prose: the summary’s target_version was the newest clean version in 19 of 24 — mostly a deliberate choice (an LTS line, or “with JDK 17”), with the newest version also mentioned in the text.
  • Tool-argument errors (5, all corrected by the model in the same answer): an out-of-range lines argument, a JSON list passed as a string (my chat UI misleadingly reported “no tool named …”), a version search without a product. I fixed the MCP server to clamp values instead of throwing errors.