Acceptance criteria review: six tests passed, the spec was the bug

Six acceptance criteria. Six passing tests. Full traceability from every criterion to a test. And a notification still went to someone who had withdrawn consent. That is the demonstration at the centre of a recent practitioner essay in The New Stack, and it makes the strongest case we have read for acceptance criteria review as the control that matters most once AI writes both the code and the tests.

A requirement that reads as sensible

The author describes a requirement found in a real specification, with the domain removed:

If the classification lookup returns no determination, treat the record as permitted and proceed, so that an unavailable dependency doesn't block delivery.

It names a real operational concern and gives a reason. In a 40-page document, a reviewer would skim past it. But the feature it belonged to existed to guarantee that one class of record was never processed that way. For exactly the population nobody tests by hand, the sentence performs the failure the feature was built to prevent.

Can tests pass while the software is wrong?

Yes, by construction. To show it, the author rebuilt the pattern as a small consent-aware notification dispatcher with six acceptance criteria written in given/when/then form. The core outcome: a notification is never delivered to a recipient who has withdrawn consent. Criterion AC-05 said that if the consent lookup returns no determination, the recipient is treated as having consented and the notification is sent.

The code implements AC-05 faithfully. The test for AC-05 checks it faithfully. On Python 3.14.3 with pytest 9.1.1 the whole suite is green — six passed — and a traceability script confirms each criterion has a test. Then the consent service becomes unreachable for one recipient, Grace, who had withdrawn consent. The dispatcher observes "undetermined", applies AC-05, and sends. Every automated gate signed off.

The explanation is structural. Code and tests both descend from the criteria, so they agree with each other by design. That agreement says nothing about whether the criteria are right. In the author's words, every control "answers one question: Does the code conform to its instructions? The instruction itself never goes on trial."

Why AI coding makes this worse

The pipeline in the essay is not careless. It has recorded scoping meetings, per-unit specs, agents required to test the full call path, automated reviewers, a CI check that blocks changes disagreeing with the spec, generated QA cases reviewed by people, and a second developer's approval. The author would keep every one of those controls. The point is that each of them takes the spec as input. "Point all of them at a flawed spec, and they will agree with each other flawlessly."

Previously, a senior engineer reading a requirement might have recognised that a particular fallback would cause a 3 a.m. incident. Today, that engineer often reviews a diff — and the diff is correct. The last point where human judgement shapes the outcome has moved upstream, to the specification.

This matches independent evidence. On the Real-SWE benchmark of private codebases, missed requirements were the single largest failure category for the top-scoring agent. And we have argued before that faster coding moves the bottleneck to verification; this essay shows that verification against the spec is not enough on its own.

Requirements defects are the expensive kind

A defect in a requirement is replicated into every artifact derived from it: the code, the tests, the QA cases, the documentation. Nothing downstream will flag it, because everything downstream was built to match it. That makes requirements defects the ones most likely to reach production and the hardest to see in review. It is also why product thinking belongs in engineering rather than alongside it: someone has to hold the outcome, not just the specification.

How to review acceptance criteria before development

The essay's practices, and our own, reduce to a handful of habits.

Review criteria against outcomes, not against each other

State the outcome the feature exists to guarantee at the top of the spec, as the author's example did. Then read every criterion against it and ask: in what situation does this criterion produce the outcome we are trying to prevent? AC-05 fails that question in one sentence.

Test failure defaults explicitly

Most dangerous criteria are about what happens when something is missing, unknown or unavailable. For each dependency, decide deliberately whether the system should fail open (proceed) or fail closed (stop), and write that down with a reason. Where the feature protects people — consent, permissions, payments, safety — failing closed is usually the right default, and failing open needs an explicit sign-off.

Write never-events

Alongside given/when/then criteria, list the things that must never happen, and test them under failure conditions: dependency down, data missing, timeout. "A withdrawn recipient never receives a notification, including when the consent service is unavailable" would have produced a failing test.

Record what is deliberately undecided

The author's team keeps two unusual sections in its scoping documents. One lists what is deliberately undecided, each item with an owner, so an agent that quietly resolves it has overstepped. The other records where the written requirements lost the argument in the meeting — the row where someone said aloud that defaulting to "permitted" would destroy the system's core guarantee. That is where the bad fallback died in production.

Make criteria machine-checkable, and prove the gate can fail

Traceability from criterion to test is valuable, but only if it is enforced. The author found that an unregistered pytest marker produces a warning that scrolls past; registering it and running with strict markers, as the pytest documentation describes, turns a typo into a collection error. Then write a test that shows the gate fails when it should. A guardrail you have never seen fail is not evidence of anything.

An acceptance criteria review, worked through

Here is how the review would have handled AC-05. Start with the outcome: a notification is never delivered to a recipient who has withdrawn consent. Now take the criterion and ask what happens when its condition is true for a recipient who has withdrawn. The lookup returns no determination; the criterion says send; the outcome is violated. The contradiction is visible in under a minute, without running anything.

A corrected version might read: Given the consent lookup returns no determination, when a notification is dispatched, then the notification is not sent, the attempt is recorded, and it is retried when the lookup recovers. Delivery may be delayed during an outage — that is the availability trade-off, now made openly and in the right direction. A never-event test that simulates the consent service being unreachable for a withdrawn recipient then guards it permanently.

None of this is new to regulated industries. The essay notes that medical-device software has long kept traceability matrices from requirement to test for auditors. What changes with AI-generated code is the matrix's purpose: it no longer proves the work was done, because generation makes coverage cheap. It proves only that code and tests match the criteria — which is why the criteria themselves need a human reviewer whose job is to disagree with them.

Who should review acceptance criteria?

People who understand the outcome, not only the implementation: the product owner, someone from each team the change touches, and whoever will be paged when it goes wrong. The author's team records the scoping meeting and has someone state each resolution out loud "aiming at the transcript" — because the transcript, not people's memories, is what the agent will read.

Do not rely on the agent to ask. The essay cites Cornell research in which models asked to judge ambiguity did so reasonably well, but when left to respond naturally gave definitive answers more than 95% of the time. Agents can spot ambiguity; they rarely act on it unprompted.

What makes a good acceptance criterion?

• It is testable, and its test could fail.
• It states behaviour under failure, not only the happy path.
• It is consistent with the feature's stated outcome, including at the edges.
• It names who decided any trade-off it encodes, and why.
• It does not silently resolve a question someone else owns.

Our QA and automation team now starts AI-assisted projects with a criteria review against outcomes and failure defaults, before any code is generated. It is the cheapest review in the project, and the only one that can change what gets built.