ornith-1.5 35B MoE @CVE triage
A 35B mixture-of-experts with only 1.9B active parameters decodes at 84 tok/s and answers in 39 s — but its score drops to 0.83, with consistently unsafe upgrade advice. My CVE prompt actually made it slightly worse.
- Tested
- Setup
Radeon Pro W7800 48 GB — bare-metal workstationBare metal, Linux, ROCm (Ollama) · AMD Ryzen 5 7600 (6C/12T, AVX-512) · 64 GB DDR5-6400
- Runtime
- Ollama 0.34.0, 262K context; CVE MCP server 0.5.2
- Models
ornith-1.5:35bQ4_K_Mtested
Qwen3.8-27B GSQ-RCOISTA IQ3_Sbaseline
Qwen3.8-27B AggressiveQ8_K_Preference
Verdict
Speed doesn't make up for it: in CVE triage, a 'safe' version that's still vulnerable costs more than waiting. ornith's strengths — zero hallucinated CVEs, good format discipline — might suit workflows where tools verify correctness, but not this one.
Works
- 83.7 tok/s decode, 39 s per answer (1.5× ISTA, 3.7× HauhauCS Q8)
- Zero invented and zero memory CVEs; KEV list 1.00; summary format 44/45
Doesn’t
- 7/24 upgrade recommendations still vulnerable — systematic (Spring4Shell 3/3, Django 3/3)
- Version tool used on only 17/24 upgrade questions
- My ISTA-tuned prompt lowered its score (0.87 → 0.83)
- Overall score
- 0.83
- Unsafe upgrade advice
- 7/ 24
- Decode
- 83.7tok/s
- Seconds per answer
- 39s
Upgrade recommendations
Out of 24 upgrade answers
- Newest clean
- Still vulnerable
Data
| Newest clean | Still vulnerable | |
|---|---|---|
| ISTA IQ3_S | 23 | 0 |
| HauhauCS Q8_K_P | 22 | 1 |
| ornith-1.5 35B | 16 | 7 |
Prompt effect on ornith
Generic 825-byte prompt vs my 3,311-byte CVE prompt
- Generic prompt
- CVE prompt
Data
| Generic prompt | CVE prompt | |
|---|---|---|
| Overall score | 0.87 | 0.83 |
The question
Does ornith-1.5:35b keep its speed advantage (89.7 tok/s advertised, twice ISTA’s) without giving up accuracy? It’s a MoE — 35.5B total, 1.93B active, Q4_K_M — and I ran it on the W7800 (100% GPU, 262K). Same prompt (byte-identical to ISTA’s through a shared YAML anchor), same sampling, MCP server and questions; I froze the truth before the run. 15 × 3, zero errors.
Results
| ISTA IQ3_S (4070 Ti SUPER) | HauhauCS Q8_K_P (W7800) | ornith-1.5:35b (W7800) | |
|---|---|---|---|
| Overall | 0.98 | 0.97 | 0.83 |
| Upgrade: newest clean / vulnerable | 23 / 0 | 22 / 1 | 16 / 7 |
| Version tool used | 24/24 | 23/24 | 17/24 |
| Fix versions correct / wrong / missing | 40/0/2 | 40/0/2 | 33/3/6 |
| KEV status correct | 6/6 | 6/6 | 5/6 |
| Memory / invented CVEs | 0 / 0 | 0 / 0 | 0 / 0 |
| Non-Turkish answers | 0 | 3 | 2 |
| Summary block | 43/45 | 44/45 | 44/45 |
| Tool calls / seconds per answer | 3.2 / 60 s | 4.2 / 145 s | 4.6 / 39 s |
| Decode | ~43.5 tok/s | 18.0 tok/s | 83.7 tok/s |
What I make of it
- The speed is real; the accuracy isn’t. 39 s per answer and 83.7 tok/s, but the score goes from 0.98 to 0.83.
- The loss is all in the upgrade advice: 7 of 24 still vulnerable (ISTA 0, HauhauCS Q8 1). And it isn’t random — Spring4Shell 3/3, Django 3/3, jinja2 1/3. For a security tool, “upgrade to X” when X is still exposed is the most expensive mistake there is.
- Shallow tool use. Plenty of tool calls (4.6 per answer), but it used the version tool on only 17 of the 24 upgrade questions. Calling more tools isn’t the same as researching deeper.
- Its strengths: no invented or memory CVEs, KEV list 1.00, prioritisation 3/3, non-existent CVE 3/3, format discipline 44/45. It keeps hallucination under control; where it falls down is reasoning about which version is actually clean.
My prompt made it worse
ornith’s first run used my chat UI’s generic default prompt by mistake, because I hadn’t added a model-specific entry yet. That run doesn’t count for the comparison, but it taught me something:
| Generic prompt (825 B) | CVE prompt (3,311 B) | |
|---|---|---|
| Overall | 0.87 | 0.83 |
| Version tool used | 22/24 | 17/24 |
| KEV status correct | 6/6 | 5/6 |
| Non-Turkish answers | 0/45 | 2/45 |
| “unverified” markers | 0 | 15 |
I’d tuned the CVE prompt on the Qwen family, and on ornith it backfires — it multiplies the hedging markers and lowers how often the model calls the right tool. But the 7/24 unsafe upgrades show up in both runs, so that failure is the model’s, not the prompt’s.