AI coding agent productivity moves the bottleneck to review

In OpenAI's own field report on coding agents in scientific research, seven of the eight projects examined still depended on people to determine whether the agents' work was scientifically correct.

That is the number worth planning against, and it is not the one being quoted. AI coding agent productivity gains are real, and they arrive attached to a constraint that most adoption plans do not budget for: somebody has to decide whether the output is right, and that somebody does not get faster.

The bottleneck moves, it does not disappear

Speeding up one stage of a process does not speed up the process. It relocates the queue.

Delivery is roughly: decide what to build, design it, implement it, verify it, integrate it, ship it, support it. Agents compress implementation dramatically. Every other stage runs at the speed it ran before, and verification now receives more work per unit time than it ever has.

One contributor to the report described the shift plainly — the role moves from implementation toward verification and orchestration. Specifying what to build, defining how correctness will be measured, deciding when something is ready. That is a different job from writing the code, and it is a job most teams have not staffed or trained for, because until recently it was a fraction of a senior engineer's week rather than the whole of it.

This is not a new observation about systems, only a new place for it to apply. Any pipeline has one constraint at a time, and optimising anything other than the constraint produces no throughput at all — it produces inventory. Right now, in most teams adopting agents, the inventory is unreviewed pull requests.

Why review does not scale the way implementation did

Three properties make verification stubborn.

Reading unfamiliar code is slower than writing familiar code. This was true before agents. The difference is that an engineer reviewing a colleague's work shares context — the conventions, the constraint that made an odd choice necessary, the conversation from Tuesday. Agent output arrives without any of that, and reconstructing intent from a diff is the expensive part.

Speed and depth come apart. One contributor to the report put it as: with coding agents it is quite easy to go fast; to go far in science there is still a need for expert guidance, understanding, taste and care. Those four words describe things a reviewer supplies, none of which are produced by the thing generating the work.

The practical version for a software team: output arrives without the uncertainty signals a colleague would give you. A person who is unsure hedges, asks, or flags the part they guessed at, and reviewers use those signals to decide where to look closely. Without them every diff demands uniform scrutiny, which is precisely the workload that does not scale.

Volume defeats care. A reviewer facing three thoughtful pull requests reviews three pull requests. The same reviewer facing thirty reviews perhaps five and approves the rest on vibes. This is not a discipline failure; it is what happens to any queue when arrival rate exceeds service rate. The output is a codebase nobody has actually read.

What this means for how you staff

The instinct is to buy more agent seats. The constraint says otherwise.

Senior engineers become more valuable, not less. The scarce skill is judging whether something is correct and appropriate, which is precisely what seniority is. If your adoption plan assumed agents would let you hire fewer seniors, it has the ratio backwards — you need proportionally more of them, doing a job that looks less like building.

Review capacity is a number you should know. How many changes per week can your team genuinely review to a standard you would defend? Almost nobody can answer. Until you can, you are scaling input against an unmeasured limit, and the symptom will present as a quality problem rather than a capacity one.

Correctness definitions become an artefact. If verification is the constraint, then anything that makes verification cheaper is leverage. Tests that actually discriminate, property-based checks, deterministic reproductions, clear acceptance criteria written before the work — these stop being hygiene and start being the throughput lever.

Measuring the ceiling

If review capacity is the constraint, it is worth measuring rather than estimating. Three numbers, over a fortnight, from data you already have.

Time from ready-for-review to first substantive comment. Not to approval — to the first comment that shows somebody read it. This is the queue depth in disguise, and it grows before anything else visibly breaks.

Approvals per reviewer per week, against diff size. If approvals hold steady while diff size climbs, the review is getting shallower. Nobody will report this and it is the leading indicator of the quality incident.

Change failure rate, split by author. Agent-assisted changes against human-authored ones. If they diverge, you have a verification gap with a size attached to it, and a number you can take to a planning conversation. If they do not, you have evidence the process is holding and can push input higher.

These are the same signals DORA has been pointing at for years, applied to a constraint that has moved. Nothing new is needed to see it — you just have to look at the review stage rather than the deploy stage.

What we do differently

We build software for clients, increasingly with agents in the loop, and three changes came out of this rather than out of a methodology.

Acceptance criteria are written before an agent touches anything, in a form that can be checked rather than admired. "Handles errors gracefully" is not a criterion. "Returns 409 with the existing resource when the idempotency key has been seen" is one, and it can be verified in seconds instead of argued about in review.

We size work so a single change stays reviewable. The temptation with agents is to ask for the whole feature, because it can produce the whole feature. A 2,000-line diff is not reviewed by anyone, whoever wrote it.

And we treat unexplained confidence as a signal to look harder. When output arrives with no caveats on the part that should have caveats, that is the section to read line by line.

The number nobody has

There is a widely circulated figure about agent-workdays per human workday inside OpenAI, and per-researcher inference spend running into hundreds of dollars a day. We could not confirm either against the primary source, which returned an error to us, so we have left them out rather than repeat them.

The seven-of-eight finding does not need them. Whatever the multiplier on output turns out to be, the constraint is the same, and the constraint is the part you can act on.

Where agents genuinely remove work rather than move it

The argument above is about changes that need judgement. A useful amount of engineering does not.

Mechanical migrations across many files, test scaffolding, boilerplate for a well-specified interface, one-off scripts that are read once and thrown away, and translations between formats are all work where correctness is cheap to establish — usually because a compiler, a type checker or an existing test suite checks it for you rather than a person.

That is the real distinction, and it is not seniority or complexity. It is whether verifying the output costs less than producing it. Where a machine can check the work, agents compound. Where only a person can, they queue.

Sorting your backlog on that axis is a more useful exercise than any agent evaluation, and it takes an afternoon. The tasks on the cheap-to-verify side are where to push adoption hard and stop worrying about review capacity entirely.

The honest position on adoption

None of this is an argument against coding agents. It is an argument against the business case that says headcount goes down.

What we see is throughput up and the composition of the work changed: less typing, considerably more specifying and checking. Teams that plan for that get the gain. Teams that plan for fewer engineers get a review queue, a quality incident about six months in, and a difficult conversation about a productivity claim.

If you are putting agents into a delivery process now, measure review throughput before you increase input. It is a week of instrumentation and it tells you where your ceiling actually is. We are happy to walk through what that looks like on a real pipeline, and if you want the longer version of how we think about this, our writing on AI engineering and on why projects fail covers the same ground from the delivery side.