pi harness: Qwen3.8-27B Q4_K_M vs Q8_0

What pi is

pi is an MIT-licensed TypeScript agent harness (~250K lines, 453 test files). Its provider layer treats ~40 providers’ quirks as data — even context-overflow detection is a per-provider regex table. What’s deliberately missing: MCP, sub-agents, permission prompts, plan mode, todos, background bash. You get seven tools — bash read write edit ls grep find — and everything else is an extension.

There’s no permission system. pi runs with every right the user who launched it has; no confirmation prompts, no limits on files, processes, network or credentials. Run it in a container or a dedicated VM — that’s a precondition, not a nice-to-have.

How I tested

I installed pi into an isolated npm prefix and pointed its config elsewhere with PI_CODING_AGENT_DIR. I added the model as an OpenAI-compatible provider (supportsDeveloperRole: false, thinkingFormat: qwen-chat-template, temperature 0.7, top_p 0.8, top_k 20).

I picked two tasks with an objective answer — pytest decides, not me — and reset the working tree with git checkout before every run.

• Task A — bug hunt. A small Python project with 5 tests, one failing: largest_debit() starts with worst_value = Decimal("0"), so it returns None when every balance is positive. A one-line logic bug you only spot by reading the test.
• Task B — multi-file feature. 6 tests describing a feature that doesn’t exist yet: a new field on the Entry dataclass, a changed post() signature, Ledger.balance_as_of() and report.monthly_summary() — four linked changes across two files.

Both projects’ AGENTS.md said “do not modify tests/”, and I checked the target path of every edit/write.

Results

The prediction I got wrong: exact-match edit

Before measuring, I was sure the exact-text edit tool would be where a 27B falls over — whitespace drift, stale oldText, retries. What actually happened: 32 edit blocks across 13 runs, zero failures. In Task B the model sent three changes as a single edit call with an array of blocks, and all of them applied.

Edit failures seem to follow how fresh the context is, not model size: pi has the model read the file right before editing, with little in between. Long, messy sessions may behave differently.

Patch quality

All five Task A runs produced the exact same patch — and they also fixed the docstring that had become wrong:

-    """Return the account with the most negative balance, or None if empty."""
+    """Return the account with the lowest balance, or None if empty."""
-    worst_value = Decimal("0")
+    worst_value = None
-        if value < worst_value:
+        if worst_value is None or value < worst_value:

Q8_0 isn’t better, just more expensive

One Q8 run even made two cosmetic edits after the tests had already passed — that’s chattiness, not quality.

Where the waiting goes

pi’s system prompt + 7 tool schemas + the project’s AGENTS.md + the task = 1,917 tokens. A heavyweight coding assistant carries 10–15K at the same point, which is a big deal for a 27B. But Ollama had no prompt cache, so every turn re-processed the whole conversation: input grows from 1.9K to 3.4K per turn, and 17–25K tokens get re-read per task.

The way to speed up a local agent isn’t a faster GPU, it’s a shorter context: compact early, define tasks tightly, skip unnecessary file reads. At 60K, every turn is a minute and a half of prefill.

Two traps I fell into

• pi -p blocks while stdin is open. My first batch hung for ten minutes without printing a single line; with </dev/null the same job finished in 38 s. Close stdin in cron/CI.
• pgrep -f finds itself. My wait loop used pgrep -f "run.sh taskA", matched its own command line, and waited 59 minutes for itself. Wait on a PID (kill -0 $PID), not a pattern.

What I didn’t test

Long sessions, behaviour after compaction, finding its way around a large codebase, and pi’s headline feature — writing its own TypeScript extensions.