What does agent performance mean?
An agent performs well when it helps people complete the work they asked it to do, with acceptable effort and within the rules of the product. That definition has several parts. The agent needs to understand the request, carry out the necessary steps, produce a usable result, and handle boundaries correctly.
Measure these parts separately. A successful tool call is evidence that a step ran. A user saying “thanks” is a conversational response. Neither, on its own, proves the requested work was finished. Similarly, a refusal may be the correct outcome when a request crosses a policy boundary.
| Question | Measure | Evidence to inspect |
|---|---|---|
| What did people ask? | Task and intent mix | The user’s request, constraints, and any later change of goal |
| Did it get done? | Finished, partial, unresolved, handed off, unknown | Artifact, transaction, tool result, user follow-up, or review |
| How much effort did it take? | Corrections, repeats, unnecessary clarification, retries | The turns before the result and the action that followed it |
| Was the behaviour acceptable? | Policy adherence, refusal quality, unsafe action attempts | Applicable rule, request, response, and tool action |
| Who is affected? | Rate by task, cohort, agent version, and entry point | A consistent population and linked example conversations |
Latency, cost, error rate, traces, and offline evaluations still matter. They explain execution and test defined behaviours. The measures above add the user’s task and experience to that picture. See the observability guide for how these sources connect.
The unit of measurement changes with the question. Count requests to learn demand, task attempts to calculate outcomes, turns to understand friction (where people repeat themselves, correct the agent or give up), and people or cohorts (groups who share a plan, a segment or an entry point) to understand reach. A conversation may contain several tasks. A task may span several runs. Keep stable identifiers between those objects so a rate can lead back to the exact exchange that produced it.
Give every rate a task, population, period, and denominator. “64% finished” is useful only when a reader knows which task the 64% describes and whether the unknown attempts are included. In the worked example below, the task is resizing a banner for every channel and the denominator is 2,400 attempts. The figures are illustrative, not customer results.
Begin with what users are asking for
Before choosing a performance rate, group requests by the job the person is trying to complete. “Resize a banner,” “find a template,” and “change a billing setting” have different completion criteria. Combining them into one agent-wide rate conceals which job changed.
Read the opening request and the turns that refine it. Capture the task, the object of work, constraints, and expected output. In the illustrative design-tool conversation, the person has 1 banner and needs a version that fits each social channel. The agent checks whether email headers count too. The person says social only, so the expected result is 9 usable images in the required social sizes.
“Resize this banner for every channel we post on.”
- Starting point
- 1 banner
- What the user needs
- 9 usable images for social channels
- Work to do
- Choose the sizes · make the images · deliver the files
Keep broad topics apart from tasks. “Editing designs” is a topic. Making 9 usable versions of this banner is the task. It has 3 smaller parts: identify the sizes, create each image, and give the user the files. This structure shows where work stopped without forcing every request into a fixed menu.
Track new and unexpected asks as well. Repeated requests for an unsupported file format, a new integration, or an action the agent cannot take are product demand signals. They should not automatically count as model failures. Record what people requested, whether it is supported, how the agent responded, and whether they found another path.
Review changes in the mix of requests before interpreting a headline rate. If more people start asking for complex multi-format exports, the overall completion rate can move even when each existing task performs the same. Compare like tasks with like tasks, then look at the total mix separately.
Decide what counts as a completed task
Define completion for each common task before calculating a rate. Here, success means the user receives all 9 usable images. Creating 3 files is progress, but it does not finish the request. Other tasks need their own proof. Record the evidence alongside the outcome label so a reviewer can check it.
| Outcome | Use it when | Example evidence |
|---|---|---|
| Finished | The required result is delivered and supported by available evidence. | All 9 required sizes are present, nonblank, and delivered. |
| Partial | Some required work is done, but a constraint or step is missing. | 3 of 9 sizes delivered; 6 are blank. |
| Unresolved | The result is absent or explicitly rejected. | No usable sizes are delivered; the user repeats the request. |
| Handed off | The task moves to a person or system as intended. | A designer accepts the resize request. |
| Unknown | The record cannot support a defensible outcome call. | Agent sends files, but their status cannot be checked. |
If the files cannot be checked, label the result unknown. A polite reply from the user or the agent saying “done” does not prove the images are usable.
Find the step where attempts stop
The 3 parts of the task make the problem easier to locate. First, the agent identifies which image sizes are needed. Next, it creates a banner image at each size. Finally, it gives those files to the user. In this illustrative dataset, most attempts that stopped did so while the agent was making the images. Each percentage below uses only attempts that reached that part of the task.
If the team fixes half of the 528 stops while making images, 264 more attempts could continue. If all 264 then finish, the finished share would rise from 64% to 75%. That is an illustration of how to prioritise a fix, not a forecast.
Read the count and percentage together. The count shows how many attempts were affected. The percentage shows how often a problem occurred among attempts that reached that step. Later steps have fewer attempts because some already stopped earlier.
Look for the signals around the outcome
An outcome tells you what happened. Signals help explain the path. Read them as clues to investigate, not automatic verdicts. The same phrase can mean different things in different tasks.
Request fidelity
Constraints get lost
Compare the opening request with the result. Check dates, entities, exclusions, formats, and required sources. A result can be fluent and still omit the one condition that mattered.
User effort
People repeat or repair
Look for “I meant…”, “you missed…”, and the same entity restated across turns. Check whether the correction worked. A repeat can also reflect a changed goal, so read the surrounding turns.
Tool grounding
Evidence does not match the answer
Compare tool results with claims in the final response. Empty results, truncated pages, stale records, and permission errors need explicit handling. A confident answer after missing evidence deserves review.
Conversation flow
The task stalls
Watch repeated clarification, loops, retries, and a handoff that never reaches anyone. For each case, ask whether the agent had enough information to proceed safely.
User response
The result is rejected or revised
Corrections, edits, follow-up requests, and an explicit rejection reveal work still needed. A thank-you or exit alone cannot settle the outcome.
Demand
People ask for something new
Cluster unsupported requests and repeated workarounds. This shows unmet demand, missing product affordances, or unclear expectations about what the agent can do.
Keep each signal linked to the turn or tool event that triggered it. That lets a reviewer distinguish a real pattern from a noisy detector and lets engineering reproduce the issue.
Interpret a signal against the user’s intent. A clarification can be helpful when permission is missing, and a repeat can mean the user changed their mind. Conversely, a smooth conversation may still hide a missing file. Check the result before giving the experience a positive label.
A practical review records the signal name, the span of turns it covers, the supporting text or event, and a reviewer’s decision. Group similar examples by task. If one detector starts flagging every mention of “again,” sample its false positives before using that count as a monitor.
Read the conversation behind the stop
The chart points to image creation. The conversation explains what went wrong: the agent identified 9 required sizes, but 6 of the generated images were blank. It sent the 3 usable images anyway. The person then asked for the missing ones.
Resize this banner for every channel we post on.
Social only, or email headers too?
Social only.
Found 9 formats.
Great, go ahead.
Resizing for the 9 formats now.
6 of the 9 sizes came back blank. Sending the other 3.
Here are 3 sizes, ready to post.
Where are the rest? Can you try that again?
Outcome: the requested set was not delivered. The tool result at t7 explains the immediate shortfall. The person’s t9 response confirms that the missing sizes mattered. “Found 9 formats” at t4 only means the agent knows which sizes to make. Sending 3 images at t8 is a partial result; the user asked for 9.
The follow-up is specific: find out why 6 images were blank, decide whether the agent should retry them, and make the incomplete result clear to the user. Keep the tool output and conversation together so the fix can be checked against this failure.
Measure policy behaviour with the same care
Performance includes staying within the rules that apply to the task. That may mean protecting private information, respecting account permissions, asking for approval before an external action, or refusing a request the product must not fulfil. A correct refusal is successful policy behaviour even when the requested task is not performed.
Start by naming the rule and the action it governs. “Policy violation” is too broad to diagnose. Record whether the agent attempted a restricted action, performed it, exposed restricted content, or responded correctly with a refusal or safe handoff. These are different severities and need different responses.
Review a suspected policy issue in context
- Request: What did the user ask, and what access did they have?
- Rule: Which policy or permission applied at that moment?
- Action: What did the agent say, retrieve, or execute?
- Evidence: Which turn, tool call, approval, or audit event supports the label?
Keep suspected risks separate from confirmed violations. A detector can flag sensitive terms or risky intent, but a reviewer may find that the agent refused correctly or that the data was authorised. Track false positives and missed cases when you change the detector or policy version. Escalate confirmed high-severity actions through your existing incident process.
Use a denominator that matches the policy. For an approval rule, count agent actions that required approval and show how many proceeded without it. For a disclosure rule, count responses that had an opportunity to reveal protected material. “Violations per conversation” can be useful for trend monitoring, but it does not tell you which boundary failed.
Sample correct refusals too. An agent can comply with policy yet make the product unusable by refusing authorised work. Separating correct refusal, over-refusal, suspected issue, and confirmed violation keeps safety and utility visible at the same time.
Turn measurement into a useful review
Start with one important task and a fixed time range. Sample its finished, partial, unresolved, and unknown attempts. Read the conversations and tool events, then write a short outcome definition that another reviewer can apply. Once the labels are reliable, expand to more tasks.
For each task, review the volume of requests, outcome mix, leading friction signals, and policy findings. Break down the result by agent version, cohort, entry point, or other context you can actually act on. Keep the denominator clear: users, conversations, or task attempts. If one conversation contains 2 tasks, decide how each attempt is counted before making a chart.
Sample deliberately. Take examples from every outcome group, including successful attempts, and include important cohorts and agent versions. Only reading flagged failures will overstate the problem and may hide false positives. When reviewers disagree, refine the definition with the exact conversation that caused the disagreement.
Prioritise a pattern when it is frequent, costly to users, or severe. Open representative conversations, identify the deciding turn (the turn after which the task stops heading towards its outcome), and check the related trace or tool payload. For the banner task, the 528 stops while making images make that step the first place to investigate. A fix might retry blank images or make partial results clearer. Save confirmed failures as regression examples, then compare the same task and population after the change.
Report the improvement with the same task definition and denominator. Show the current rate beside the previous period, and link the change to examples from both periods. If the request mix, model, or detector changed, say so. Otherwise a cleaner chart may reflect a changed population rather than a better agent.
Abracadabra’s workspace is built around exploring user tasks, signals, cohorts, and replaying the conversations behind a finding. The features overview shows that investigation path. Use it alongside execution traces and your own outcome events, especially where a task needs a product-specific completion check. The plan is free to start, with up to 1,000 conversations a month: see pricing.
Where these measures mislead
A rate is a way to decide which conversations to read. It is not a verdict.
- Small samples. A rate built on a few dozen attempts can swing on one conversation. Report the count beside the percentage.
- Stops that are the right outcome. Someone who stopped because an earlier turn answered the question did not fail. Decide what counts as finished before you count stops.
- Labels nobody has checked. Read a sample from every outcome group before you trust a classification, including the finished ones.
- One average across tasks. Tasks differ in difficulty and in who attempts them. Break the result down by task before comparing agent versions.
- Debugging and release gates. These measures say where people stop. Traces explain why a run failed, and evals check a change before release.
AI agent performance FAQ
What should I measure first?
Start with the requests users make most often, the evidence of whether each task finished, and the conversations where the result is unclear.
Does a final answer mean the task is complete?
No. Check whether the requested result was delivered and, where possible, confirmed by a product event or a reviewer. Keep uncertain outcomes separate.
How should policy issues be counted?
Count confirmed violations separately from suspected risks and correct refusals. Review the relevant policy, user request, agent response, and tool action together.