What is AI agent observability?
AI agent observability should tell you what the agent did and whether the person finished. It connects execution data, evaluations, safety events, and the outcome of the conversation.
That last clause matters. A conventional service usually tells you it failed with an exception, a timeout, or a bad status code. An agent can finish every call, return HTTP 200, and still misunderstand the request, choose the wrong tool, omit half the source material, loop through clarifications, or produce an answer the person cannot use.
You need three views of the same interaction.
The production evidence stack
three questionsTraces, spans, prompts, responses, tool calls, state transitions, handoffs, latency, tokens, cost, retries, and errors.
Correctness, groundedness, policy checks, regression tests, human review, and release comparisons.
Task completion, intent-level stops, repetition, clarification, abandonment, cohort differences, and the exact turn underneath each pattern.
Most AI agent observability tools start with execution and evaluation. Those views help engineers debug runs and validate releases. Customer-facing agents also need conversation analytics and replay.
The short version
- Use traces to reconstruct what the system did.
- Use evaluations to test whether behaviour meets a defined standard.
- Use conversation analytics and replay to see what users tried, where tasks stopped, and which people or cohorts keep hitting the same failure.
- Connect all three with stable conversation, user, version, and task identifiers so an aggregate can lead back to evidence.
A green run can still end with a person leaving empty-handed.
Why AI agent observability is different
Traditional monitoring assumes that useful behaviour is mostly deterministic: the same input follows a known path, failures produce recognisable telemetry, and a request is a reasonable unit of analysis. Agents break all three assumptions.
| Dimension | Traditional service | AI agent |
|---|---|---|
| Execution path | Mostly known in advance | Branches across models, tools, memory, retries, and handoffs |
| Failure signal | Exception, timeout, 4xx or 5xx | Often a plausible response with no technical error |
| Unit to inspect | Request or transaction | Run, conversation, task, user, and cohort |
| Quality | Usually implied by successful execution | Must be assessed separately from execution |
| Change surface | Code and infrastructure | Code, model, prompt, retrieval, tools, memory, policy, and data |
| Useful evidence | Logs, metrics, traces | Telemetry plus evaluations, conversation context, user outcome, and replay |
The failure modes are different too
A tool errors, a model call times out, or the orchestration loop crashes. Traces and alerts should catch it.
The agent returns a fluent but wrong, incomplete, or unsupported answer. Evaluation or human review is needed.
The agent completes some intents and quietly skips another. The run looks successful; the user's task is not.
The overall rate looks stable while one plan, region, agent version, or returning-user cohort gets materially worse.
Nothing breaks at once. More people repeat themselves, abandon, or require clarification after a model, prompt, or data change.
A single quality score hides the work. It can show that a detector moved. It cannot show which task changed, where people stopped, or what they said. Keep the path from the score to the conversation intact.
AI agent tracing: what the run can tell you
A trace is the ordered record of a run. It connects the initial request to model calls, retrieval, tool invocations, memory operations, retries, handoffs, and the final response. In a multi-agent system, it should also preserve which agent delegated work to which other agent and what context crossed that boundary.
Use spans to answer concrete engineering questions: Where did the time go? Which tool failed? Which prompt and model version ran? How many tokens did the step consume? Did a retry change the result? Which state transition caused the loop?
If the agent is down, slow, expensive, or looping, start with the trace.
Here is one attempt at summarise what my team shipped, read twice.
6 of the 9 documents came back empty. summarising the other 3.
this is missing half of it.
In the illustrative run above, the trace records 12 spans, 9 tool calls, no errors, and no timeouts. Retrieval returned 9 times. The software did what it was asked to do.
But 6 of those returns were empty. The agent summarised the other 3 documents and presented the answer as complete. On turn 9 the person said, “this is missing half of it.” The task stopped. The trace is accurate. It is also incomplete: it cannot tell you the person left empty-handed.
What a useful agent trace should preserve
- Identity and lineage. Trace, span, conversation, user, workflow, agent, and parent-agent identifiers.
- Version context. Model, prompt, tool, policy, retrieval index, and agent version.
- Execution events. Model generations, tool inputs and outputs, memory reads and writes, retries, handoffs, and human interventions.
- Operational measures. Latency, tokens, cost, status, error class, and resource use where relevant.
- Outcome join keys. The stable identifiers that let a run connect to a task outcome, evaluation result, and conversation replay later.
Do not treat hidden chain-of-thought as a required observability field. Capture the actions, state changes, tool decisions, inputs, outputs, and explicit rationale your system is designed to expose. That is more reliable, safer to retain, and sufficient for most investigations.
OpenTelemetry already names the run
You do not have to invent a schema. The OpenTelemetry generative AI semantic conventions (opens in a new tab) already name the operations an agent performs and the attributes each one carries. They are still experimental, so expect movement. Use them where they fit. A shared schema also makes a later vendor change easier.
- Operations.
gen_ai.operation.nametakesinvoke_agent,invoke_workflow,chat,execute_tool,retrieval,create_agent,embeddings,generate_contentandtext_completion. - Who ran.
gen_ai.agent.id,gen_ai.agent.name,gen_ai.agent.version,gen_ai.request.modelandgen_ai.response.model. - What it did.
gen_ai.tool.name,gen_ai.tool.call.id,gen_ai.tool.call.arguments,gen_ai.tool.call.resultandgen_ai.response.finish_reasons. - What it cost.
gen_ai.usage.input_tokensandgen_ai.usage.output_tokens, plus the cache and reasoning token counts. - The join key that matters most.
gen_ai.conversation.id. It is the one attribute that lets a span find its way back to the exchange it came from.
The standard does not name the task, its steps, or whether the person finished. Add those fields yourself.
What AI agent observability should measure
A production scorecard should cover four questions: Is the service healthy? Is the agent behaving acceptably? Are users completing meaningful tasks? Can the team get from a bad number to the conversations that explain it?
| Layer | Measure | What it tells you |
|---|---|---|
| Operations | End-to-end and step latency | Where users wait and which model or tool owns the delay |
| Operations | Error, timeout, retry, and loop rate | Whether execution is breaking or becoming unstable |
| Operations | Tokens and cost per task | Whether a useful outcome is becoming more expensive |
| Evaluation | Correctness, groundedness, and policy checks | Whether outputs meet a defined test or review standard |
| Experience | Task completion and stop rate | Whether people got the outcome they came for |
| Experience | Intent-level drop-off | Which step inside the task creates the most loss |
| Experience | Repetition, clarification, and abandonment signals | Where the conversation is creating friction without throwing an error |
| Experience | Cohort and version spread | Who is affected and what changed around the same time |
Define completion before you measure it
Task completion is not the same as a positive final message. An agent can say done after skipping a step. A person can also get what they need without explicitly confirming it. Define the outcome for each important task before turning it into a rate.
Use a small set of states that the team can review consistently:
- Finished. The requested result was delivered and the available signals support completion.
- Stopped. The conversation ended before the result was delivered, or the person explicitly rejected it.
- Partial. Some of the task was completed, but a required intent was missed.
- Handed off. The agent moved the task to a person or another system as designed.
- Unknown. There is not enough information to make a defensible call.
Keep explicit events separate from inferred signals. A completed transaction, saved document, or accepted handoff is stronger than an LLM judgement based on the final turn. Inference still matters when no product event exists, but it should carry a detector version and confidence. Review a sample of each outcome regularly, especially after changing the model, prompt, taxonomy, or detector.
The denominator matters too. Report how many attempts reached each intent, not only how many conversations started. Otherwise a late step can look healthy simply because an earlier step lost most people first.
Start with a real task, not a generic “agent quality” score. Count every conversation where somebody asked to summarise what my team shipped. In the illustrative dataset used throughout this site, there are 2,400 attempts: 1,536 finished and 864 stopped.
2,400 attempts at one task
illustrativeThe 64% completion rate is a baseline, not a diagnosis. The task holds three smaller intents:
- Find the source. Which documents count as what the team shipped. 92 stops.
- Read it. Open each document and pull out what is in it. 528 stops.
- Write the summary. Turn what came back into something they can send on. 244 stops.
The largest absolute loss and the worst step rate both sit on read it: 528 of the 2,308 attempts that reached that step stopped there, a 22.9% step failure rate. Halving that loss would move another 264 attempts through the task, lifting completion from 64% to 75% in this illustrative example.
An actionable metric names the task and the step. It also opens the conversations underneath.
There is no universal “good” completion rate across every kind of agent. Compare the same task across periods, versions, and cohorts. Then inspect the change underneath the number.
The right data model for agent observability
A span cannot represent a person's outcome. A dashboard average cannot explain one. Connect them without losing either.
From execution to evidence
connected units- Topic groups the broad subject area, such as product questions.
- Task is what the person came to accomplish. It is the outcome worth measuring.
- Intent is one step inside that task: find the source, read it, write the summary.
- Signal is something notable on a specific turn: repetition, clarification, tool trouble, dissatisfaction, or recovery.
- Person joins behaviour across conversations instead of treating every session as a stranger.
- Cohort compares people by plan, region, behaviour, version, or metadata you already send.
The same schema works across research agents, support agents, and in-product assistants. Each produces tasks, intents, signals, people, and cohorts. The labels come from the conversations, so you do not need to design every branch first.
Keep the original words. Put the summary above them and the fields beside them. Features covers how each of the six is detected and what you can ask of it.
A practical investigation workflow
Good observability tells you what to fix first. Start with the pattern, then narrow it to the conversations and runs behind it.
Pattern to proof
one drill pathStart from completion, stops, repetition, or another production signal.
See which step creates the largest loss and the worst rate.
Separate a global problem from one concentrated in a plan, region, or returning-user group.
Open the conversations behind the aggregate and identify the repeated mechanism.
Compare like with like. Confirm improvement without moving the failure elsewhere.
In the running example, the intent breakdown puts read it first. The transcripts show 6 empty documents. The trace identifies the retrieval operation that returned them. The cause could sit in retrieval, prompt behaviour, permissions, or source quality. Now engineering knows where to start.
Each team gets the part it needs. Product sees the scale and affected users. CX sees the conversations and accounts. Engineering gets run identifiers and spans. Evaluation turns confirmed failures into regression cases. Keep these as links in one chain, not screenshots in four tools. The workspace shows this drill path.
Turn production failures into tests
Production and evaluation should form a loop. Conversation analytics finds a repeated task failure. Replay shows the turns where it happens. The trace identifies the run, tool, model, and version involved. Once the team confirms the cause, that conversation becomes the basis for a regression case.
Do not copy sensitive production text into a test set by default. Redact it, preserve only the details needed to reproduce the mechanism, and record why the case belongs in the dataset. A good regression case includes the expected task outcome, the critical intent, the relevant tool behaviour, and the failure condition it is meant to catch.
After the fix passes the eval, return to production and watch the same task, intent, version, and cohort. An offline pass shows that the known case improved. It does not show that completion improved for everyone or that the failure did not move to another step.
How to implement AI agent observability
Start with one task and trace it end to end. Collecting every field before you have a review workflow adds storage, privacy risk, and cost.
Choose one production task
Pick a frequent, consequential job with a recognisable finish. “Summarise what my team shipped” is better than “general assistant quality.”
Define observable completion
Write what finished, stopped, partial, and handed-off mean. Use user-visible evidence when possible; do not make one LLM judge the only source of truth.
Instrument the run
Record hierarchical spans for models, tools, retrieval, memory, handoffs, and human interventions. Attach versions and stable join keys.
Capture conversation context safely
Preserve the turns needed to understand the task. Redact sensitive fields, restrict access, set retention, and avoid collecting content you do not need.
Connect task, person, and cohort
Join conversations across time and attach useful metadata such as plan, region, agent version, and release. Keep the original transcript reachable.
Build the review loop
Review the largest intent-level losses, read a bounded sample, route confirmed causes, add regression cases, and compare the same task after the fix.
Minimum useful event contract
| Keep | Examples | Why |
|---|---|---|
| Identifiers | trace, span, run, conversation, user, agent | Connects the same event across tools and levels |
| Versions | model, prompt, tool, workflow, policy, index | Explains what changed before a regression |
| Timing and status | start, end, latency, status, retry, error class | Supports operational monitoring and trace reconstruction |
| Interaction | turn role, tool name, safe input/output reference, handoff | Shows how the run and conversation unfolded |
| Outcome | task, intent, finished or stopped, signal, reviewer label | Connects execution to the user-visible result |
| Context | plan, region, release, custom metadata | Reveals concentrated failures hidden by the average |
Here is the contract on the tool call that came back empty. OpenTelemetry names the yellow fields. You add the blue fields to connect the run to the person's outcome.
One span, both halves
illustrative{
"gen_ai.operation.name": "execute_tool",
"gen_ai.agent.id": "team-assistant",
"gen_ai.agent.version": "4.2.0",
"gen_ai.conversation.id": "c_88421",
"gen_ai.tool.name": "search_documents",
"gen_ai.tool.call.id": "call_04",
"gen_ai.usage.input_tokens": 1840,
"gen_ai.usage.output_tokens": 96,
// not named by the standard. you add these.
"app.topic": "product questions",
"app.task": "summarise what my team shipped",
"app.intent": "read it",
"app.turn.index": 7,
"app.result.count": 0,
"app.task.outcome": "stopped",
"app.person.id": "p_4471",
"app.person.plan": "enterprise",
"app.person.region": "emea"
}You do not need to paste a second analytics snippet into every turn to use Abracadabra. Connect the conversations you already retain, live or backfilled. The product reads topic, task, intent, and signal from the conversation, then lets you compare people and cohorts using the metadata attached to them.
How to choose AI agent observability tools
Choose a tool for the investigation your team needs to run. Feature counts are a poor proxy for that job.
Choose by the question
complementary layers| Category | Best for | What to require, and what it will not replace |
|---|---|---|
| APM and agent tracing | Debugging runs, latency, errors, cost, and execution paths | Require hierarchical spans, OpenTelemetry export, prompt and tool context, version tags, and multi-agent lineage. It does not replace user-outcome analytics or a disciplined eval program. |
| Evaluation platform | Testing releases and scoring sampled production outputs | Require datasets, human review, custom evaluators, experiment comparison, and regression gates. It does not replace operational tracing or whole-population product analytics. |
| Conversation product analytics | Tasks, intents, friction, users, cohorts, trends, and replay | Require automatic classification, drill-down to turns, cohort comparison, version joins, and no forced taxonomy. It does not replace run-level debugging or UI replay outside the conversation. |
| Security and governance | Access, policy, audit, data handling, and risky actions | Require redaction, retention, permissions, audit history, policy events, and intervention controls. It does not explain quality, product outcome, or engineering cause by itself. |
Common tools and the questions they answer
These tools answer different questions. A team may need more than one.
| Tool | Built to answer | What it leaves open |
|---|---|---|
| LangSmith | Why a run behaved this way. It brings traces, datasets, and evals together, especially for LangChain or LangGraph teams. | Task outcomes across the full population. Runs alone do not explain what someone tried to finish over 9 turns. |
| Langfuse | Run-level tracing with an open-source, self-hostable, OpenTelemetry-based product. | Tasks, intents, and cohorts still need to be defined and derived. |
| Arize Phoenix | Whether output quality holds up, with tracing and an evaluation library. It is also OpenTelemetry-based and self-hostable. | Whether the person finished. A good score on sampled outputs does not prove task completion. |
| Braintrust | Whether a release performs worse than the last one, using datasets, experiments, and CI gates. | Production behaviour outside the dataset. An eval only tests the cases included in it. |
| Datadog, Grafana | Agent telemetry alongside infrastructure monitoring, alerts, and on-call workflows. | What the conversation meant and whether the person completed the task. |
| Abracadabra | What people asked for, which step stopped them, who it hit hardest, and the turns underneath. Product analytics and session replay for the conversation. | Run-level debugging and release gating. Keep a tracer and an eval tool for those. |
Pricing changes too often for a durable comparison here. Check each vendor's current page against the volume you send. Abracadabra's plans are on one page.
What to check before you buy
- Can an aggregate open the exact runs or conversations behind it?
- Can you filter by agent, prompt, model, tool, release, user, task, and cohort without rebuilding the schema?
- Does the product distinguish technical success, evaluated quality, and user task completion?
- Can you control redaction, retention, access, sampling, and export?
- Can confirmed production failures become regression cases and can fixes be compared on the same task?
- Will product, CX, engineering, and AI quality teams use the same evidence chain, or create separate screenshots?
AI agent observability FAQ
What is AI agent observability?
AI agent observability is the practice of collecting and connecting execution telemetry, evaluations, safety events, and user-outcome evidence so a team can understand how an agent behaves in production and investigate failures.
Is Abracadabra an observability tool?
Abracadabra is an observability platform for the conversation: what people asked for, what worked, and what broke. Traces tell you the run finished. Abracadabra tells you whether the person did. Keep tracing for run-level diagnosis and evals for defined quality checks.
How is agent observability different from LLM observability?
LLM observability often centres on individual model calls: prompts, completions, tokens, latency, and evaluations. Agent observability follows the larger execution graph, including tools, memory, state, retries, handoffs, and the outcome across a multi-turn task.
What is the difference between monitoring and observability?
Monitoring tells you whether known signals crossed a threshold. Observability preserves enough connected evidence to investigate questions you did not predict when the system was instrumented.
Does this replace LangSmith, Langfuse, or Braintrust?
No. Keep traces for run-level diagnosis and evals for defined quality checks. Abracadabra reads the conversation for tasks, intents, signals, people, cohorts, and the turns underneath each pattern.
What should I measure instead of a generic success rate?
Track operational health, evaluated quality, task completion, intent-level stops, conversation friction, cost per useful outcome, version changes, cohort spread, and the evidence behind each metric. Do not collapse them into one score.
Do I need to capture chain-of-thought?
No. Capture observable actions, tool decisions, state changes, inputs, outputs, versions, explicit rationale your system is designed to expose, and user-visible outcomes. Hidden reasoning is not a required telemetry field.
When is Abracadabra the wrong tool?
Use tracing first when the agent is erroring, timing out, looping, or unexpectedly expensive. Use an eval platform when you are testing a labelled dataset or release. Use UI session replay when the question is whether people can find or operate the surrounding interface. Abracadabra becomes useful once there are real conversations to read.
Is the 2,400-attempt dataset real customer data?
No. It is illustrative, and it is used consistently throughout this post.