AI agent consistency testing: run it five times

Run your agent eval suite once and you get a pass rate. Run each case five times and you get a different, much less comfortable number. IBM researchers measured exactly that gap: 77% of individual runs passed, but only 53% of tasks passed all five consecutive attempts. AI agent consistency testing is the difference between those two figures, and the gap was 24 points.

The second number is the one your users experience, because a customer does not get an average. They get one run, on one afternoon, and either it worked for them or it did not.

What was measured

The paper is Closing the Consistency Gap: Self-Evolving Agents That Learn to Stay on Course, submitted on 8 September 2026 by Evelyn Duesterwald, Benjamin Elder, Lilian Ngweta, Shashanka Ubaru and Malgorzata Zimon.

The setup is deliberately ordinary: a ReAct agent on the AppWorld benchmark using GPT-4.1. Every task was run five times, and two metrics recorded — the average pass rate across individual runs, and the fraction of tasks that succeeded in all five. The first came out at 77%, the second at 53%.

Nothing exotic produced that gap. It is what happens when you stop averaging.

Worth being clear about what the paper does and does not claim. It is a benchmark result on one agent architecture with one model, not a universal constant — your own gap could be narrower or considerably wider depending on how many steps your tasks take and how tightly your tools are specified. The transferable finding is not the 24 points. It is that the gap exists, that averaging conceals it, and that it is cheap to measure on your own system.

Why a small divergence becomes a different outcome

The mechanism is worth understanding because it explains why this is structurally worse for agents than for ordinary software.

An agent's run is a sequence where each step conditions the next. One slightly different tool call at step three produces a different observation at step four, and by step eight the trajectory has diverged completely. The agent is not making one probabilistic decision, it is making twenty in a chain, and small variation compounds rather than averaging out.

This is also why turning the temperature down does not fix it. Lower temperature narrows the distribution at each step; it does not remove the branching, and a single divergence early is enough to reach a different end state. Teams that have set temperature to zero and still see inconsistency are not misconfigured — they are meeting the structure of the problem.

It is the same reason evaluation has to score the trajectory rather than the final answer. What this paper adds is that scoring the trajectory once is still not enough. You have to score it repeatedly and look at the variance.

AI agent consistency testing in one change

The change to make this week costs nothing but compute. Run each eval case five times and report two figures side by side: the mean pass rate, and the proportion passing every single run.

Then treat cases that pass three or four times out of five as a distinct category. Not green, not red — flaky, and blocked from release. This is the part teams resist, because a case passing four times out of five feels like a pass. It is not. It is a coin weighted in your favour that you are about to flip, repeatedly, in front of paying customers, and at any scale worth having, a four-in-five case fails for a fifth of everyone who encounters it.

The rule is familiar from ordinary test suites, where a flaky test is treated as a broken test. Agents make it harder to ignore because the flakiness is not in the harness. It is in the product.

There is a second reason to record both numbers rather than replacing one with the other. The mean tells you how good the agent is; the all-run figure tells you how reliable it is, and they move independently. An agent that improves from 70% to 80% mean while its consistency figure stays flat has usually become better at easy cases and no more dependable on hard ones — which is the opposite of what a production rollout needs. Watching the two together is what stops a model upgrade from looking like progress when it is not.

Finding where the runs split

Once you are running five times, the diagnostic falls out almost free. Record the tool-call sequence for each run and compare them. The step at which the sequences first differ is the unstable decision, and it is almost always a specific one rather than general drift.

In our experience the divergence point is usually one of a few things: an ambiguous tool description where two tools plausibly fit, a prompt that does not say what to do when a lookup returns nothing, or a step where the agent must decide whether it has enough information to proceed. All three are fixable with clearer instructions, and none of them is a model problem.

That is what the researchers automate. Their framework pairs a Consistency Analyzer, which identifies the unstable steps in a trajectory, with a Guideline Generator, which turns each diagnosis into a targeted guideline held in episodic memory for later attempts. The reported result is a 16 point rise in tasks succeeding across all five runs on same-task evaluation, and 13 points on similar-task generalisation.

Read that as validation of the manual version. If writing a specific guideline at the point of divergence works well enough to move the metric that far automatically, it works when a person does it too.

Cost is the obvious objection and it is smaller than it looks. Five runs is five times the eval spend, but eval suites are typically a rounding error next to production inference, and the alternative is discovering the variance through customer reports. If the suite is genuinely too expensive to run five times, that is a signal the suite is too large rather than a reason to keep averaging — twenty cases run five times is more informative than a hundred run once.

What this changes commercially

The gap has a direct consequence for how agent projects should be scoped and sold, and it is uncomfortable.

A demo is a single run. A pilot is a handful of runs, usually observed by people willing the system to succeed. Production is thousands of runs by people who are not. A system demonstrating at 77% and consistent at 53% will look impressive in the room and disappoint in the quarter, and nobody involved will have been dishonest.

So the acceptance criterion should be the consistency figure, agreed before the build. For anything touching money, permissions or customer data we would want all-five-run success at or near 100% on the critical paths, achieved by narrowing what the agent may do rather than by hoping for a better model. For advisory or drafting work where a person reviews the output, a lower bar is defensible because the human is the consistency mechanism.

That distinction — who absorbs the variance — is the real design question, and it is worth settling before any code is written. It also pairs with the operational side: once you know your consistency number, session traces tell you which runs were the failures and why.

There is a hiring and process implication too. Someone has to own the consistency number and be willing to block a release on it, and on most teams that person does not currently exist — evals are owned by whoever built the agent, which puts the author of the system in charge of the criterion their own work is judged against. Separating those two roles matters more here than in ordinary software, because the failure is statistical rather than binary, and a motivated author can always produce a run that worked and call the rest bad luck.

None of this requires the framework in the paper. It requires running the suite five times instead of once, reporting the harder number, and refusing to ship the flaky cases. That is a day of work on an existing eval suite and it is the first thing we would do on any agent project heading for production — the same standard we apply to quality engineering generally, and the reason AI engineering work needs a test discipline rather than a demo.