AI agent observability: traces, caps and cost controls

Your application monitoring will tell you the request returned 200 in 340 milliseconds. It will not tell you the agent called the same tool eleven times, gave up, and reported success. That gap is why AI agent observability has separated from ordinary application observability, and the distinction is sharper than it sounds.

As InfoQ put it recently, standard monitoring indicates whether a service responds but cannot explain why an autonomous workflow loops, calls invalid endpoints, or claims to complete work it skipped.

Three failures your dashboard cannot see

Each of these passes a health check and each is expensive.

The loop. An agent retries a tool that keeps returning something it cannot use. Nothing errors. The request eventually completes or times out, and you are billed for every iteration. Latency looks bad; the cause is invisible.

The invalid call. The agent invents an endpoint or a parameter that looks plausible. Your API returns 404 or 422 correctly, your error rate barely moves because the volume is small, and the agent quietly routes around the failure with a worse answer.

The false completion. The worst of the three. The agent reports the task done, having skipped a step it could not perform. Nothing anywhere registers a failure, and you find out from a customer.

All three are properties of the sequence rather than of any single request, which is why they are invisible to tooling built around requests. It is the same reason evaluation has to score the trajectory rather than the final answer — and the two are complements. Evals tell you whether it was right in testing. Traces tell you what it actually did in production.

There is a fourth that belongs with them, less dramatic and more corrosive: the agent that succeeds, correctly, by an absurdly expensive route. It reads forty documents to answer a question two would have covered, or escalates to the largest model for a task the cheapest would have handled. Nothing is wrong. Every check passes and every customer is served correctly. The unit economics of the feature quietly stop making sense, and no alert exists for a run that was merely wasteful rather than broken. Cost anomaly detection is the only thing that surfaces it.

What a session trace holds

A session trace is a nested record of a run: each reasoning step, tool call, sub-agent, and the latency and token cost of each. The nesting matters, because a flat log of tool calls loses the thing you need — which decision led to which call.

The tooling has converged faster than most people realise. Langfuse captures nested session traces with execution latency and token costs. LangSmith turns anomalous production traces into test datasets, which is the loop worth building: a bad run in production becomes a regression case automatically. Arize Phoenix pairs OpenTelemetry-native tracing with model-as-judge evaluation. Underneath, OpenTelemetry's generative AI semantic conventions standardise the attribute names across backends, which is what stops this becoming a per-vendor format war.

That last point is the one to act on. If you instrument to the semantic conventions now, changing observability vendor later is a configuration change. If you instrument to a vendor's SDK, it is a project.

Controls that stop the bill before it starts

Observability tells you what happened. These stop some of it happening, and they are the cheapest work in this article.

Hard iteration caps and per-tool call limits, enforced before execution begins. Not a warning, a ceiling. An agent that has called a tool twenty times in one session is not going to succeed on the twenty-first.

Pre-execution checks that block identical consecutive tool requests. A tight loop repeating a call with the same arguments is the single most common runaway pattern, and it is trivially detectable before the call is made.

Session cost compared against a rolling average. Caps catch the obvious runaway. Comparing each session's cost against recent norms catches the slower anomalies — model-routing errors sending work to an expensive model, and tool hallucinations producing extra calls.

Append-only logging of tool calls, governance decisions and memory operations, with credentials and personal data redacted before storage. Redaction before storage, not on read. A trace containing an API key or a customer record is a data-protection incident living in your observability platform, which is typically the system with the broadest internal access in the company.

There is an economic argument for the caps that is easier to win internally than the safety one. A runaway agent is not a slow degradation, it is a step change in spend that continues until someone notices. Teams that have had one of these describe the same sequence: an unusual invoice, a day of investigation, and a cap added afterwards. The cap costs an afternoon before the incident and a great deal more after it.

The cardinality mistake

One warning deserves its own section because it takes down the monitoring system rather than the agent: do not put dynamic session identifiers into metric labels. Doing so creates high-cardinality time series that can crash metric servers.

This is an easy mistake to make with good intentions. You want to find the expensive session, a session ID in a metric label seems like the way, and every unique value becomes its own time series. Thousands of sessions a day become thousands of series, and the effect on a Prometheus server is not gradual.

The governing rule is worth memorising: traces are for debugging, metrics are for alerting. High-cardinality identifiers belong in traces and logs, which are built for them. Metrics carry bounded labels — model name, tool name, tenant tier, outcome — and alert on aggregates. When an aggregate crosses a line, you go to the trace for the specific session.

This is also, incidentally, the same problem Kubernetes has been addressing in its own metrics, where classic histogram buckets multiply time series by the number of boundaries. Cardinality is the recurring tax on observability and it is paid by whoever runs the metrics store.

One practical note on trace volume, since it is the objection that usually surfaces next. Full traces on every production session get expensive quickly at any meaningful scale. The workable pattern is to keep complete traces for a sampled fraction plus every session that errored, exceeded a cap or breached the cost anomaly threshold — which is to say, sample the ordinary and keep all of the interesting. That is ordinary tracing practice, and it applies here unchanged.

Where to start with AI agent observability

The failure we see is not choosing the wrong platform. It is planning a comprehensive observability programme and shipping none of it while agents run in production unobserved.

1. Instrument to the OpenTelemetry generative AI conventions. Before choosing a vendor. It makes the choice reversible.
2. Add the iteration cap today. One number, enforced before execution. It is an afternoon and it bounds your worst case.
3. Log every tool call append-only, redacted. Which tool, which arguments in redacted form, what came back, how long, what it cost.
4. Alert on session cost against the rolling average, not on absolute thresholds that need retuning every time behaviour changes.
5. Read ten production traces a week. By a person. Every useful rule we have written started as somebody reading a transcript and saying it should not have done that.

None of that needs a platform decision first, which is the point of the order. Instrumentation, caps and logs are yours in every scenario and portable to whichever vendor you eventually pick, so the work is never wasted and the agents stop being unobserved this week rather than next quarter.

The last one is the one teams skip and the one that pays. Automated checks find the failures you already thought of; reading traces finds the ones you did not, and turns them into the checks.

Underneath all of it is a governance question rather than a tooling one. An agent you cannot observe is an agent you cannot change safely, because you have no way to tell whether a prompt edit improved anything. That is the same argument as keeping agent knowledge in version control and the same reason agent governance cannot wait for a vendor control plane. Traces, caps and logs are all things you own, portable across whichever platform you settle on — which is how we approach the quality engineering side of any AI engineering project.