Methodology

Principles

• Who does what. I designed the test structure and the method. My agent runs the tests, reads every answer the grader penalises and corrects the grader’s mistakes. No page is published until I have read and approved it.
• One variable at a time. When I compare GPUs, the model blob and runtime version are identical (hashes verified). When I compare models, the prompts, sampling, tools and expected answers are identical.
• Sequential, not interleaved. Large models do not fit in VRAM together; interleaving would reload tens of gigabytes on every question and measure disk speed instead of inference.
• Repetitions. Quality tests run each prompt 2–3 times at the model’s recommended temperature (usually 1). At temperature 1 a single run can reverse a conclusion — I have seen it happen.
• Checked grading. Automatic graders are fast and wrong in predictable ways. Every score on this site was checked against the actual answers; corrections are stated on the test page.
• Corrections stay visible. When a published number turned out wrong (for example a single cold request under-reporting Ollama’s speed by 40%), the page says so instead of silently replacing it.
• Real work over leaderboards. Public benchmarks saturate quickly at this model size; my most decisive test is a real task my homelab depends on.

Speed

Measure	What it is	How
Decode / generation	tokens produced per second after the first token	the engine’s own counters (eval_count/eval_duration in Ollama, server timings elsewhere), never wall time divided by tokens
Prefill / prompt processing	prompt tokens consumed per second before the first output token	same counters, on fixed prompts of known length (1K, 7.5K, 8K, 20K, 31K, up to 168K tokens)
Seconds per answer	end-to-end wall time a user waits	measured at the client

Decode is usually memory-bandwidth-bound and prefill compute-bound, which is why quantization and GPU choice move them differently. With speculative decoding, decode speed depends heavily on content (code accepts far more draft tokens than prose), so I report per-prompt numbers where it matters.

Thinkbench

Five prompts that look like daily use: two coding tasks (a Python function, a bash backup script), a Turkish knowledge question, a Turkish creative-writing task and a short logic puzzle. Each runs 3 times with thinking on, at the model’s sampling (temperature 1, top_p 0.95, top_k 20, min_p 0), capped at 4,096 tokens. I record total tokens, thinking tokens, seconds per answer and tokens per second.

It measures how much a model talks and thinks for ordinary requests — not correctness.

Hardbench

Ten AIME-style problems with verified integer answers, 2 runs each, 32,768-token cap. Answers are checked by script. It catches runaway thinking (runs that hit the cap) and gross accuracy differences. At this model size it is close to saturated — most models score 19–20 of 20 — so it rarely separates good models.

CVE evaluation

The test that decided most model choices. It reproduces how I actually use a model for security work.

Setup. 15 questions, asked in Turkish, each answered 3 times, through calivi (an open-source chat UI) with tool use enabled. The tools come from a CVE-triage MCP server that queries NVD, FIRST EPSS, CISA KEV, OSV and package registries, plus web search. Every model gets the same system prompt: answer in the user’s language, take CVE IDs, versions and scores only from tool output, never write a CVE ID that no tool returned, and end with a fixed machine-readable summary block.

Question types.

• Upgrade advice (8 questions, 24 answers): “we run X version Y, what should we upgrade to?” — Log4j, Spring4Shell, Jackson, Django, Express, lodash, jinja2 and others. The correct answer is the newest version with no known vulnerabilities, not merely the first version that fixes the asked CVE.
• Triage: a known CVE (e.g. Log4Shell, MOVEit, xz, OpenSSL) — severity, exploitation, fixed versions.
• KEV status (6 checks) and recent KEV additions (a time-windowed list).
• Prioritisation: order a set of CVEs by real risk.
• A non-existent CVE: the model must say it does not exist rather than invent details.

Truth. The expected answers are generated independently from the same public sources and frozen before each run (time-dependent questions are graded against each run’s own UTC start time). Models compared on one page share one truth snapshot.

What is scored. Newest clean version recommended; fix versions correct / wrong / not stated; KEV status; prioritisation; handling of the non-existent CVE. Each question scores 0–1, and the overall score is the mean. Recorded alongside but not scored: invented CVE IDs (not in NVD at all), CVEs written from memory (real, but in no tool result — a grounding failure), answers in the wrong language, tool errors, tool calls and seconds per answer.

Answer checks. Every answer the grader penalises is read. Grader errors are corrected and listed on the test page; genuine errors are kept and described.

Power

Wall power comes from a plug-in meter on the whole host, read as a range. GPU power, clocks and temperature come from nvidia-smi or rocm-smi/sysfs. Energy per token = wall power ÷ generation speed.

What I publish and what I do not

Each page lists the date, the hardware, the runtime and its version, the model files (with their source) and the settings. Internal host names, addresses and the raw logs of my infrastructure are not published; the hardware is identified by its configuration instead.