Same Model. Four Harnesses. What Actually Survives the Verifier?
A 40-trial FeatBench study of CritiqueCode, Claude Code, Oh My Pi, and OpenCode running the same Mercury 2.5 preview model in fresh E2B sandboxes.
Critique
Research study
Same model. Four harnesses.
What survives the verifier?
critique.sh / research
Agent harness · BYOA · Composer 2.5
Across 40 canonical trials, the four harnesses resolved 28 tasks. CritiqueCode led resolution at 8/10. Claude Code was fastest by median agent runtime at 1:03. Every harness passed every easy task; the separation appeared in the hard block.
This study started with one constraint: do not accidentally benchmark four different models and call the result a harness comparison. Mercury stayed fixed at inception/mercury-2.5-preview, routed through OpenRouter. Harbor owned the trial and evaluator. E2B supplied a clean machine. The harness was the moving part.
The benchmark, drawn
The harness is the variable.
Four views tell the story: resolution, task-level separation, the cost/time tradeoff, and work volume. The chart grammar follows Lieflat Charts; the values are the canonical V1 records.
CritiqueCode led resolution.
One rung = one task/harness trial resolved · n = 10
The hard block made the gap.
Filled dot = full F2P + P2P pass · 10 tasks × 4 harnesses
Speed and cost are separate wins.
X = displayed cost · Y = median agent minutes
Tool volume did not predict resolution.
Each tick = 10 normalized tool calls · label = failed tools
The short read: resolution, speed, and cost split apart
CritiqueCode produced the strongest observed resolution/cost combination in this sample. Claude Code finished fastest by median agent time. Oh My Pi and OpenCode used substantially more tokens, but work volume alone did not predict success. These are descriptive findings from one run per pair—not a universal ranking.
Why CritiqueCode belongs in the comparison
CritiqueCode is Critique’s author agent: an interactive coding session that implements the requested change, then hands the work through a forced review and verified-repair loop. The author does not get to certify its own work by saying “done.” That makes CritiqueCode a particularly relevant harness for this study: its product thesis is that implementation and confidence should be separate stages. The benchmark still grades it by the same external FeatBench verifier as every other harness.
Author, then review
An interactive author agent for making the change and handing it off to the review system.
The author is not the judge
Completion claims route into the existing review machinery instead of ending the workflow.
Fixes earn promotion
Repair is followed by verification, so a passing result—not narration—decides whether work is finished.
In this first sample, that approach also produced the highest observed resolution (8/10) and the lowest displayed cost per resolved task. Those numbers are evidence worth investigating, not a universal victory lap: the task block is small, one run was made per task–harness pair, and the shared provider credential limits cost certainty.
| Harness | Resolved | Displayed cost | Median time | Tokens |
|---|---|---|---|---|
| CritiqueCode | 8 / 10 | $0.236291 | 1:29 | 5,584,876 |
| Claude Code | 7 / 10 | $0.537726 | 1:03 | 15,609,408 |
| Oh My Pi | 6 / 10 | $0.373359 | 1:16 | 16,671,057 |
| OpenCode | 7 / 10 | $0.684547 | 1:07 | 17,082,376 |
The easy ceiling was real. The hard block was the study.
All four harnesses resolved the five easy tasks: SmolAgents 783, Starlette 2806, PyJWT 913, Tox 3288, and Dynaconf 1295. That 20/20 block is a useful floor, but not a useful separator. The five harder tasks produced 8/20 resolutions and exposed where context, repair, tool use, and termination behavior diverged.
| Task | Critique | Claude | OMP | OpenCode |
|---|---|---|---|---|
| DSPy 7964 | 2/2; 208/208 | 2/2; 208/208 | 0/2; 206/208 | 1/2; 208/208 |
| Mesa 2296 | 10/10; 204/204 | 8/10; 203/204 | 0/10; 204/204 | 10/10; 204/204 |
| OpenAI Agents 508 | 29/29; 186/186 | 29/29; 186/186 | 29/29; 186/186 | 29/29; 186/186 |
| aiogram 1594 | 1/3; 717/717 | 1/3; 717/717 | 1/3; 717/717 | 1/3; 717/717 |
| SmolAgents 1442 | 0/7; 297/321 | 0/7; 320/321 | 5/7; 320/321 | 4/7; 303/321 |
Why “done” was not the measurement
Eleven runs contained language that looked like a completion claim while the external verifier still reported failure: CritiqueCode 2, Claude Code 3, Oh My Pi 3, and OpenCode 3. The detector is only a text heuristic. The result that counts is whether the independent FeatBench verifier passed both the new-feature suite and the existing regression suite.
- Official/reference solution ran before Mercury spending.
- All ten selected tasks returned reward 1.0.
- Known deterministic task failures were excluded before freezing.
- The task list was frozen before harness trials.
- Harbor ran the external verifier after agent exit.
- F2P measured the requested feature.
- P2P measured preservation of existing behavior.
- The final diff and verifier logs were retained.
Exactly how the 40 trials ran
The launcher used two sandboxes concurrently and rotated which harness went first by task. Setup time was recorded separately from agent time. No sandbox was reused between harnesses. Claude Code was configured through its Anthropic-compatible OpenRouter path and was not silently switched to an Anthropic model when Mercury behaved differently.
What we kept besides the score
The score answers who resolved the task. The trajectory helps explain how. Every canonical record keeps the model request count, normalized tool events, failed tools, files changed, lines added/deleted, timing boundaries, provider-accounting source, termination flags, patch, agent log, and verifier log.
- 1,583 model requests across 40 trials.
- 1,850 normalized tool calls.
- 345 failed tool calls retained as telemetry.
- 183 files changed; 2,684 lines added; 1,921 deleted.
- One Claude/OpenRouter compatibility/API event.
- One Oh My Pi setup loss, rerun in a fresh sandbox.
- No canonical timeout or crash.
- Zero strict F2P-complete/P2P-incomplete regressions.
The cost number needs a footnote
The displayed aggregate is $1.831922. It is useful for directional comparison, but it is not four independent invoices. The provider reconciliation pass found 725 generation IDs: 724 responses were available and one returned HTTP 404. Nineteen canonical trials use complete provider totals. Twenty-one use the catalog estimate (input × $0.04/M) + (output × $0.15/M) from recorded tokens.
Open source means following the whole trail
The point of publishing this is not to ask you to trust a polished number. It is to make the number inspectable. The public repository contains the frozen configuration, selection rationale, task list, generated task environments, adapters, launcher, normalization/accounting scripts, oracle evidence, 40 canonical bundles, patches, chart data, and the complete technical study.
Reproduce V1
git clone https://github.com/repath500/mercury-harness-benchmark.git
cd mercury-harness-benchmark
uv tool install harbor
export E2B_API_KEY="your-key"
export OPENROUTER_API_KEY="your-key"
export HARBOR_SOURCE_DIR="/path/to/harbor"
git -C "$HARBOR_SOURCE_DIR" apply benchmarks/mercury_v1/patches/harbor-e2b-v1.patch
PYTHONPATH=. uv run --no-dev --extra e2b --project "$HARBOR_SOURCE_DIR" python3 benchmarks/mercury_v1/scripts/run_v1.py --concurrency 2
python3 benchmarks/mercury_v1/scripts/fetch_openrouter_usage.py
python3 benchmarks/mercury_v1/scripts/normalize_v1.py
python3 benchmarks/mercury_v1/scripts/build_chart_data.pyThe public bundle documents the additional CritiqueCode support patch and the exact artifact policy.
What V1 says—and what it leaves open
V1 says that changing only the harness produced different observed outcomes with Mercury held constant. It does not say CritiqueCode will beat every harness, that catalog estimates are invoices, or that these results transfer to multi-container web applications. Harbor’s E2B integration restricted this study to CPU-only, Dockerfile-backed, single-container tasks. The next serious step is replication of this frozen ten-task block with dedicated provider keys, followed by a pre-registered second block.