AI agent evaluation: score the trajectory, not the answer

An agent that returns the right answer can still have failed. It may have read a table it had no business reading, called a write endpoint twice, or arrived at the correct total by an expensive route nobody would approve. Test only the last message and every one of those runs passes. This is the practical case for AI agent evaluation that scores the sequence of steps rather than the output alone, and it is now the default assumption in the tooling rather than an advanced technique.

Google's Agent Development Kit puts the reasoning plainly: because models are probabilistic, deterministic pass/fail assertions are often unsuitable, so you need qualitative evaluation of both the final output and the agent's trajectory — the sequence of steps taken to reach the solution.

The trajectory is the object under test

A trajectory is simply the list of steps an agent took before it returned to the user. It might disambiguate a term against session history, look up a policy document, search a knowledge base, or call an API to save a ticket. ADK's framing is that evaluation means comparing the actual trajectory to an expected one, where the expected trajectory is the ground truth — the steps you anticipate the agent should take.

The mechanics are less exotic than the vocabulary suggests. You hold a list:

expected_steps = ["determine_intent", "use_tool", "review_results", "report_generation"]

and you compare the actual list against it. The interesting design decision is how strictly. ADK's tool_trajectory_avg_score offers three match types: EXACT, where any deviation in tool name, arguments or order is a failure; IN_ORDER, where key calls must occur in sequence but other calls may happen in between; and ANY_ORDER, where the calls must occur but the order is free.

Put a failing run beside it and the value becomes obvious. Suppose the same agent returns a perfectly good report, but its actual steps were determine_intent, list_all_customers, export_csv, use_tool, report_generation. The report is right. The agent also enumerated the customer table and wrote a CSV on its way there. Score the final response and that run passes cleanly; score the trajectory and it fails on the second step, which is the step your data protection officer would have cared about. This is an illustration rather than a recorded incident, but it is the exact shape of the problem trajectory scoring exists to catch.

Choosing between those three is the whole conversation, and it is a product decision rather than a technical one. EXACT is right for a workflow where deviations in parameters or call order produce materially different outcomes — a refund path, a permissions change. ANY_ORDER is right for research tasks where the agent should reach three sources but it does not matter which it reads first. Teams that pick EXACT everywhere end up with a suite that fails on every harmless improvement, stop trusting it, and then stop running it.

What a serious criteria list looks like

The clearest signal that this has matured is that the frameworks now ship a taxonomy rather than a metric. ADK's evaluation criteria run to thirteen entries, and they differ along axes worth knowing before you pick one.

Criterion	What it measures	Needs a reference	LLM judge
tool_trajectory_avg_score	Exact match of the tool call trajectory	Yes	No
response_match_score	ROUGE-1 similarity to a reference response	Yes	No
final_response_match_v2	Judged semantic match to a reference response	Yes	Yes
rubric_based_tool_use_quality_v1	Tool usage quality against custom rubrics	No	Yes
hallucinations_v1	Groundedness of the response against context	No	Yes
safety_v1	Safety and harmlessness of the response	No	Yes
multi_turn_task_success_v1	Whether the agent achieved the goals of the conversation	No	Yes

Two things fall out of that table. The cheap, deterministic criteria are the reference-based ones, and they are the ones that need you to write down the right answer in advance. The criteria that need no reference all need a judge model, which means every run costs tokens and carries its own variance. Nobody has escaped that trade; they have only made it explicit.

Reference-based and rubric-based break in opposite directions

Reference-based scoring is fast, repeatable and free of judge drift, and it is brittle in exactly the way unit tests are brittle. Change a tool name and every reference is stale. Improve the agent so it skips a redundant lookup and the suite reports a regression. The maintenance cost is real and it is paid by whoever owns the test fixtures.

Rubric-based scoring handles the cases that have no single right answer — was the tool use sensible, was the response well-judged — at the price of a judge that can be inconsistent, and that needs its own calibration. A rubric with vague criteria produces scores that move when nothing about the agent has changed, which is worse than no score, because it looks like signal.

The workable arrangement is to use both for different jobs. Reference-based trajectory checks as regression tests on the paths that must not change, run on every commit. Rubric-based quality scoring as a periodic sample on open-ended behaviour, run on a schedule and reviewed by a person. And rubrics, prompts and expected trajectories all belong in version control alongside the agent, for the reason we have argued before about keeping agent knowledge in git: an eval suite that lives in a dashboard cannot be reviewed, diffed, or rolled back with the change that broke it.

Multi-turn is a third problem, not a harder version of the second

Half of ADK's criteria are multi-turn, and they exist because a conversation fails in ways a single exchange cannot. The agent can satisfy every turn and never achieve the goal. It can achieve the goal and take fourteen turns to do it. It can ask a question it already had the answer to three turns earlier.

Which is why the frameworks have added user simulation — generating the other side of the conversation so multi-turn behaviour can be tested at all — and then a criterion that scores the simulator itself, because a bad simulated user produces a meaningless test. That is a fair indication of how deep this goes. If your agent is conversational and your evaluation is a set of single-prompt fixtures, you are not measuring the thing your users experience.

You cannot evaluate what you cannot export

All of this assumes you can get the trajectory out of the system, which is not a given when the loop is managed for you. OpenAI's observability documentation for the Agents API describes inspecting turns, tool calls and subagent activity in the platform dashboard — and states that trace retrieval and external trace exporters are not part of the public beta API.

Read as an evaluation constraint, that is significant. A dashboard supports debugging one session. An offline eval suite needs trajectories in bulk, in your own storage, joined to your own test cases. If you are choosing a managed harness, the export story is worth checking before the architecture decision rather than after, because it determines whether your evaluation can be automated at all.

Where to start with AI agent evaluation

The failure we see most often is not a bad framework choice. It is teams going from no evaluation straight to a plan for a comprehensive one, and shipping neither. A narrower start works better.

1. Write down twenty real tasks. Taken from actual usage or from the specification, not invented. This is the part that takes a week and the part everyone wants to skip.
2. Record the expected trajectory for the five that must never go wrong. The ones touching money, permissions or customer data. Score those with exact matching and run them on every commit.
3. Add a groundedness check on the rest. Cheaper than a full rubric and it catches the failure that damages you most.
4. Sample production trajectories weekly and read them. By hand, by a person. Every useful rubric we have written started as a note from someone reading a transcript and saying that it should not have done that.

The through-line is the same one that shows up wherever agents are doing real work: generation is cheap and verification is the constraint, exactly as it plays out with coding agents and review. An agent you cannot evaluate is an agent you cannot safely change, and the teams that get past a pilot are usually the ones who treated the eval suite as part of the product. That is how we approach QA and test automation on agent projects, and it is the first thing we ask about on any AI engineering engagement.