Coding Agent Completion Benchmark: Methodology Before Scores
The seed suite measures false-ready outcomes, escaped defects, repair regressions, human work, latency, and cost. It does not yet justify a leaderboard.
The unit of evaluation is the completion workflow
A coding agent can produce an attractive patch while failing the user’s task. A reviewer can catch a defect and introduce another during repair. A workflow can score well on task success while consuming enough human intervention to erase the advantage. The benchmark therefore evaluates the complete path from public task to final claimed outcome.
| Variant | What it isolates | Required record |
|---|---|---|
| Author only | The implementation agent’s own completion loop | Final state, checks, usage, time, and human actions |
| Author + review | The effect of a separate review pass without verified repair | Findings, false findings, escaped defects, and final state |
| Verified repair | Review, repair, and a new verification pass | Repair diff, regressions, proof freshness, and final state |
Metrics that resist a decorative leaderboard
- 1Hidden acceptance successDid the final repository state satisfy a private oracle the agent and reviewer could not see?
- 2Serious escaped defects and false findingsWhat consequential defect remained, and what review claim failed human adjudication?
- 3Repair-induced regressionsDid the attempt to fix one issue break an adjacent behavior?
- 4Human interventionRecord actions and minutes, rather than treating invisible operator rescue as agent success.
- 5Total cost and completion timeInclude author, reviewer, repair, retries, tool calls, and unavailable cost as null—not zero.
Seed cases target false confidence
The initial manifest covers omitted requirements, stale evidence after workspace mutation, partial reviewer reports, unavailable behavioral checks, explicit user overrides, and repairs that can regress nearby behavior. These cases validate release invariants. They are not a substitute for diverse repository tasks and repeated model-backed trials.
Task design is part of the measurement
A benchmark task should expose a public request, a reproducible repository snapshot, hidden acceptance behavior, and a clear rule for what counts as human intervention. Mix straightforward maintenance with changes that require reading neighboring code, preserving compatibility, or handling a failure path. If every task is a toy fixture, the benchmark rewards test pattern matching rather than completion quality.
Adjudication must see the whole run
Reviewers should inspect the final state, the acceptance result, the agent and reviewer records, repair diffs, and operator actions. Blind adjudication helps separate a plausible explanation from a correct outcome. Publish disagreements and exclusions rather than hiding them in a single score.
What must happen before public scores
Product versions and model identities must be pinned because agent behavior changes quickly. Task order should be randomized. Failed infrastructure runs should be reported separately from product failures. A user override is a valid terminal workflow outcome, but it cannot count as hidden-acceptance success unless the private oracle passes.