Blog / · 5 min read
Coding-agent benchmarks lie less than before. We still read them wrong.
SWE-bench, Terminal-Bench, agent leaderboards: numbers are improving, but they do not tell you whether an agent will survive your codebase.
We love tables.
One model at 87%. Another at 83%. A green arrow. A curve. Three screenshots on social media. Suddenly, everyone knows which agent “wins”.
Except nobody works inside a benchmark.
We work in messy repositories, with incomplete tests, unwritten conventions, environment secrets, vague tickets, internal APIs, forgotten migrations, and a colleague who named a folder new-new-final.
Coding-agent benchmarks are useful. The problem is how we read them.
SWE-bench won, then got tired
SWE-bench had a huge merit: it pulled models away from “write me a function” and forced them to solve real GitHub issues. Read a repo, understand a bug, modify several files, pass tests. That was a healthy rupture.
But when a benchmark becomes the market’s main playing field, it wears out.
Models genuinely improve, yes. Harnesses improve too. Prompts improve. Systems around models improve. And sometimes the result measures benchmark optimization as much as general agent capability.
This is not a scandal. It is normal.
Every popular benchmark eventually becomes a product.
Terminal-Bench is interesting for a boring reason
Terminal-Bench and more environment-oriented evaluations have one advantage: they test the dirty part of the work.
Not just “find the patch”. More like:
- navigate a shell;
- install the right things;
- read errors;
- manipulate the filesystem;
- understand a failing command;
- not break the environment on the way.
It is less clean. Therefore closer to our days.
An agent that can write code but panics when bun install prints a warning is not very interesting. An agent that understands a test fails because the local fixture is stale is worth more than a model that shines on a well-framed issue.
Software is not only producing a diff. It is surviving the environment.
The missing number: review cost
To me, the best coding-agent benchmark still does not exist.
It would measure this:
how long does a competent human need to accept or reject the produced work?
That is brutal, but it is the real metric.
An agent can solve a task, pass every test, and still leave a terrible diff:
- too many files touched;
- useless abstractions;
- verbose comments;
- out-of-scope renames;
- subtly changed behavior;
- a correct migration that nobody wants to review.
On paper, it is “pass”. In your day, it is debt.
Conversely, an agent that stops at 80%, explains the blocker, keeps the diff clean, and proposes two options can be more useful than one that forces its way to the end.
Benchmarks like clean endings. Real work likes clean exits.
How to read a leaderboard
I am not saying we should ignore scores. I read them too. But I treat them as weak signals, not purchase decisions.
I look at:
Task type. Python bugfix? React frontend? DevOps? Terminal? Large repo? Agents do not all have the same muscles.
Harness. Who executes? With which tools? Which permissions? How much time? How many attempts? A good model in a bad harness can lose. A medium model in an optimized harness can shine.
Cost. A score without cost is advertising. If an agent wins with 10 times more tokens, it is not the same product.
Variance. I prefer a slightly lower but stable agent to a brilliant one that is catastrophic every other run.
Failure mode. Leaderboards show who succeeds. I want to know how it fails. An agent that fails by stopping cleanly is usable. An agent that fails by rewriting half the repo is dangerous.
Your internal test matters more
Every serious team should have its own mini-benchmark.
Not 500 tasks. Ten is enough to start.
Real tasks:
- a bug already fixed;
- a small feature;
- a migration;
- a PR review;
- a test to write;
- an anonymized production error;
- a documentation task;
- a deliberately ambiguous refactor.
Run three agents on it. Measure:
- result;
- time;
- cost;
- diff size;
- files touched;
- review time;
- human follow-ups;
- final-summary quality.
Then you learn something.
Not “which model is best in the world”. Which model is good for you.
The 2026 marketing trap
The agent war will push everyone to publish scores. That is normal. Customers want anchors. Marketing teams want a simple sentence.
But for a developer or CTO, the simple sentence is rarely the right one.
An agent at 90% on a public benchmark can be mediocre on your Symfony monorepo. A less impressive agent can understand your stack better because its tool reads logs well, respects instructions, or integrates better with CI.
The frontier is no longer just the model.
It is the pair: model + tools + context + guardrails + review ergonomics.
When you buy a coding agent, you are not buying a brain. You are buying a way of working.
Takeaway
Benchmarks are not false. They are incomplete.
They say: “in this controlled setting, with this harness, on these tasks, here is what happened”.
We translate too quickly into: “this model will code well in my repo”.
The gap between those two sentences is where bugs, bills, and ruined Fridays hide.
Read the scores. Then run your own test.
The leaderboard can tell you where to start. It cannot decide for you.
Sources
Keep reading