One Model, Seven Harnesses, 140 Trials

This is the second Mercury harness study. V1 compared four harnesses on ten tasks. V2 expands the field to seven: Pi (vanilla), Oh My Pi, Claude Code, Codex, the official DeepSeek Harness, CritiqueCode, and OpenCode. The model is held at z-ai/glm-5.3-flash through OpenRouter. Harbor owns the task lifecycle and external grading. E2B provides the disposable machine. The harness is the experimental variable.

The headline is not just who won. It is where the harnesses separate.

The cleanest quality result belongs to DeepSeek Harness and Oh My Pi: both resolved every task in this block. Pi and Claude Code were one task behind. OpenCode remained competitive at 18/20. Codex landed at 15/20, while CritiqueCode resolved 13/20. Those counts are the primary result because every “pass” means the external evaluator accepted both the requested feature and the existing regression suite.

Difficulty is a shape, not a label handed down by FeatBench

FeatBench’s task metadata calls these instances hard; it does not provide the easy / hard / very-hard distinction used here. Before any agent run, we froze ten focused tasks, five broader tasks, and five very broad tasks using static F2P count, reference-patch surface, and regression-test surface. All 20 first passed the official oracle solution. The strata are study design choices, not claims about an intrinsic difficulty scale.

Why CritiqueCode is interesting even when the score is not the highest

CritiqueCode is not a thin alias for the Critique sidecar. It is Critique’s author agent: an interactive Pi session that implements the request and is then forced through the existing review and verified-repair machinery. The point is structural: the author does not get to certify its own work by narrating “done.” The external FeatBench verifier still decides the benchmark result, exactly as it does for every other harness.

The fair reading is nuanced. CritiqueCode’s 13/20 resolution rate was below the perfect scores of DeepSeek Harness and Oh My Pi. At the same time, its median agent runtime was 2:24 and its displayed cost per resolved task was $0.005065, both the lowest observed in this run. It also produced seven completion claims that the verifier rejected—the highest count in the matrix. That combination makes CritiqueCode especially useful for studying the difference between execution efficiency, patch quality, and trustworthy termination.

What the extra telemetry says

The score alone cannot tell us whether a harness was economical because it was focused, because it stopped early, or because a provider reported usage differently. V2 therefore retains the native traces and adds a common vocabulary: READ, SEARCH, EDIT, SHELL, TEST, LSP, SUBAGENT, and OTHER. It also records first request latency, first tool latency, active-time share, context growth, tool efficiency, test-run events, patch size, and provider-ledger completeness.

The verifier is the referee

FeatBench separates new behavior from regressions. F2P tests should fail on the starting commit and pass after the change. P2P tests should continue passing. A task is resolved only when both sets pass. Agents can run whatever local tests their harness chooses, but the official result comes from Harbor’s external verifier after the agent exits.

How the 140 trials actually ran

The launcher rotated harness order by task and ran two E2B sandboxes concurrently. No harness was placed into a sandbox previously used by another harness. The task list and model were frozen before the agent matrix. Setup time was kept outside agent runtime. Human intervention was disabled. The DeepSeek Harness adapter uses the official deepseek-ai/deepseek-harness package and its headless dsh mode; Claude Code used its compatible OpenRouter route and was not silently replaced with an Anthropic model.

What broke, and how we kept the evidence honest

Harbor’s E2B artifact collector stalled after some agents had already completed. We did not rerun those agents or silently treat missing files as failures. We retained the native agent evidence, launched a fresh verifier-only sandbox with the saved patch, and marked the canonical record with a recovery flag. Seven outcomes used this path. Two recovered outcomes failed the verifier and remain failures. This is not equivalent to pretending the original Harbor process was uninterrupted, so the distinction is part of the published record.

V1 is context, not a causal before-and-after

V1 used Mercury 2.5 preview, ten tasks, and four harnesses. V2 uses GLM-5.3-Flash, a new 20-task block, and seven harnesses. It is tempting to call a percentage change a harness improvement, but the model, task mix, and execution period changed too. The responsible comparison is descriptive: V2 widens the measurement and gives us a better basis for the next preregistered replication.

Everything is open for inspection

The point of this study is not to ask you to trust a polished chart. The frozen task list, selection rationale, oracle gate, Harbor launcher, adapter code, normalizer, provider-usage fetcher, chart inputs, canonical result bundles, recovery annotations, and generated Markdown study are all in the reproduction repository. The raw harness traces are kept separate from the compact chart data so you can audit both the headline and the path that produced it.