Coding agents can be evaluated. We just have to evaluate the work.
Stop grading coding agents like chatbots
I recently argued with a software factory provider, whose position was that coding agents cannot be evaluated. Their reasoning was understandable. Software engineering is open-ended. Requirements are incomplete. Repositories contain years of undocumented decisions. Two engineers can solve the same problem in completely different ways, and both can be right. An agent may fail one run and succeed on the next. A benchmark can never reproduce all the context, negotiation, and judgment involved in shipping production-grade software.
“Coding agents are non-deterministic, operate over long horizons, and can produce many valid solutions. None of that makes them unevaluable. It means we need to stop grading them like chatbots.”
Related: AI shutdown not so simple
All true. But “difficult to evaluate” and “cannot be evaluated” are very different claims. We already evaluate traditional software systems with many possible implementations. I argue coding agents should be held to the same standard. A coding agent is not just a model. It is made up of a model, a harness, tools, repository context, instructions, permissions, an execution environment, and a feedback loop. Change any one of those and the outcome can change materially. A stronger model with poor repository context may perform worse than a smaller model with the right tools and a fast test suite. This is one reason public benchmarks are easy to misuse. A score is often described as if it measures the underlying model, when it actually measures a particular model-agent-environment combination under a particular token and time budget. When evaluating a coding agent, we are evaluating the whole system.
Evaluate behavior, not reference diffs
One objection to coding-agent evals is that exact-match grading does not work. An agent can produce a valid patch that looks nothing like the human-authored reference patch. I agree. But that just means we need to grade behavior, not compare to a reference diff. Start the agent from a known repository state. Give it a task and the context available then. Then evaluate the resulting repository against executable contracts:
- Do the existing tests still pass?
- Do hidden tests for the requested behavior pass?
- Are public APIs and data formats still compatible?
- Do migrations work in both directions?
- Are performance and resource-use limits respected?
- Did the agent modify anything outside the allowed scope?
- Did static analysis or security checks find new problems?
These checks allow multiple implementations while preserving a clear definition of acceptable behavior. They are also much harder to bluff than a prose explanation of what the agent believes it accomplished. Some qualities are not fully captured by executable tests, e.g., maintainability, architectural fit, naming, or whether a change makes the next change unnecessarily difficult. These require judgment. But human-written software has never stopped being testable because code review contains judgment.
Related: Temporal Sees AI Boost Amid Revenue Surge
Layered evaluation is necessary
“Did the tests pass?” is necessary. I argue it’s not enough in this context. An agent can make the new test pass by weakening an existing assertion. It can hard-code an expected value. It can replace a focused implementation with an enormous rewrite that happens to be correct today. It can solve the task after twenty failed attempts, consume an unreasonable budget, and leave behind changes no engineer would approve. A useful evaluation therefore has several layers:
- Outcome: Did the final repository satisfy the task?
- Change quality: Would we accept the implementation?
- Trajectory: How did the agent get there?
- Human intervention: How much help did the agent need?
- Economics: Was the result worth the financial cost?
- Production impact: What happened after the merge?
No single number captures all six layers, and that is fine. Engineering teams already use a scorecard rather than one magic metric to assess delivery.
