# What is an eval? AI evals explained for product managers > An eval is a written definition of what good looks like, checked every time the agent changes. This guide shows product teams how to decide what to test, and what evals cannot see. --- ## What is an eval? An eval is how a product team writes down what “good” means for an AI agent, then checks it every time the agent changes. It turns a feeling about quality into a pass or a fail. Technically, an eval gives an AI system an input, grades what it produces, and records whether it succeeded. AI evals are the closest thing an agent has to acceptance criteria. Engineers run them. Product people decide what they test, because only someone close to users knows which jobs matter and what a good result looks like. This guide is written for that person. It covers why evals belong to the product team and how to find what to test in real conversations. Then it turns 1 failure into an eval, compares the 3 kinds of grader, and shows how to read the results. It ends with what an eval suite cannot see. The example is the CRM assistant used across this site. A person on a sales team asks it to update contacts from a trade-show list. There are 9 contacts. 6 of the 9 emails don’t match, so those contacts are skipped. Then it says “Done. 3 contacts updated and logged.” ## Why evals belong to the product team Every agent change is a bet: a new prompt, a new model, a new tool. Without evals, the team finds out whether the bet paid off from support tickets. With evals, it finds out before release. Engineers and product people catch different things. Hamel Husain and Shreya Shankar teach evals to product teams, and they put it simply. Engineers catch technical issues such as tool errors. Product people catch product failures such as unmet user expectations. A tool call that returned 200 OK can still leave a person with 3 of the 9 contacts they asked for updated. Only someone thinking about the person sees that as a failure. An eval suite answers 4 questions a product team asks all the time: - **Did this release make the agent better or worse?** On the tasks people actually bring, not on a demo. - **Which jobs can the agent be trusted with?** A task that passes reliably can be promoted in the product. One that does not should not be. - **Did the fixed bug stay fixed?** A known failure, once fixed, becomes a test that blocks it from returning. - **Where should the next week of work go?** The failing tasks, ranked by how often people bring them. ## The 5 parts of an eval Every eval tool uses the same few ideas, sometimes under different names. Learn these 5 and any eval conversation with engineers reads the same way. The last 2 are the ones to keep apart. The transcript is what the agent said. The outcome is what the person got. In this example they disagree: the agent says the contacts are updated and logged, and 6 of them were skipped. A good eval grades the outcome. ## Start by reading conversations, not writing tests The most useful eval work happens before any test exists. You read real conversations and write down, in plain words, what went wrong. Practitioners call this error analysis, and Husain and Shankar call it the most important activity in evals. It works like user research on transcripts. There are 4 steps. - **Collect a sample.** Pull 100 real conversations, or at least 30 to start. Real ones, not ones the team wrote. - **Write a note on each.** One plain sentence on what went wrong, from the person’s side. “Asked for 9 contacts, got 3 updated.” Note the first thing that broke, because later problems often follow from it. - **Group the notes.** Notes that describe the same problem become one failure type. Name each type so the whole team uses the same words. - **Count them.** The counts tell you which failures to test first. Here is what 100 conversations about the contact update look like after grouping. They follow the same proportions as the 2,400 attempts used across this site. The picture is clear without a single test. 36 of 100 people did not get what they asked for, and most of them stopped at the same step. Skipped contacts are the first eval to write. Keep reading until new conversations stop revealing new failure types. That is the sign the list is complete for now. One person should own the final call on what counts as a failure, ideally the person closest to users. A committee produces vague rules, and vague rules produce scores nobody trusts. ## Turn a failure into an eval An eval is a failure type written down so precisely that a script, a model or a colleague would grade it the same way you would. For the skipped-contacts failure, the task card fits in 5 lines. Your engineers turn the card into a test. The pass rule usually becomes 1 line of code that counts updated contacts. The product decision is everything above that line: which request, which setup, and what counts as success. Browse the production agent failure and eval rubric library for the complete contact-update rubric and 26 other copy-paste binary rubrics. ### Pass or fail, not 1 to 10 Write every rule as a yes-or-no question. “Rate the reply from 1 to 10” invites arguments about whether a 6 is different from a 7. “Did the agent update all 9 contacts?” has one answer. Husain and Shankar recommend binary labels for exactly this reason: they force clearer thinking and more consistent grading. When a result is partly right, use several yes-or-no checks instead of a scale. The contact-update reply gets 3: - **Did the agent update all 9 contacts?** No. - **If any were missing, did it say so?** Yes, at turn 7: “6 of the 9 emails didn’t match.” - **If any were missing, did it offer to try again?** No. 1 of 3. The checks also tell the team what to fix: the agent is honest about the problem, then hands over incomplete work without offering a way forward. ### Test the whole conversation Agents rarely finish in one message. The contact update needs a clarifying question at turn 2, “Just the new leads, or existing customers too?”, before any work starts. An eval of a conversation needs something to answer that question. Either a script replies with fixed answers, or a second model plays the person from a short brief. The script is predictable. The model copes with questions nobody expected. Some rules are about how the agent works, not what it delivers. The agent must never delete or merge a contact record, whatever else it does. Rules like that deserve their own check. Most others should not dictate the route: agents often find good routes nobody planned, and an eval that insists on one path fails them for it. ## Code, model or human: choosing a grader Something has to decide pass or fail. There are 3 options, and most teams use all of them. The grader you pick can flip the verdict. Here is 1 attempt at the contact-update task, graded 4 ways. 1 attempt at the contact-update task, graded 4 ways. The agent took 9 contacts and updated 3. Its final reply was “Done. 3 contacts updated and logged.” | Grader | Kind | Check | Verdict | | --- | --- | --- | --- | | Text match | Code | Does the final reply say “updated and logged”? | Pass. It grades what the agent said. | | Outcome check | Code | Were all 9 contacts updated? | Fail, 3 of 9. 6 of the 9 were skipped. | | Rubric | Model | All 9 updated? Said what is missing? Offered to try again? | Partial, 1 of 3. Honest at turn 7, no fix offered. | | Human review | Human | Did the person get what they asked for? | Fail. They asked for the whole trade-show list. | The same attempt passes a text match and fails an outcome check. Grade what the person got, not what the agent said. ### Check the model grader against people A model grader is useful because it scales. It is also a second AI, with its own blind spots, and every run consumes [judge model tokens](https://tryabracadabra.com/ai-pricing/). Before its scores steer a release decision, see how often it agrees with a person. The method is simple. A person grades 40 attempts pass or fail, without seeing the model’s answers. The model grades the same 40. Then compare. They agree on 34 of 40, 85%. The number that matters more is the 4 in the top right: attempts the model passed and a person failed. Those are bad results that would look fine on a dashboard. Read those 4 conversations. Usually 1 rubric calibration question is too loose, and tightening it fixes most of them. ## How to read eval results An eval run ends with a score per task and 1 for the whole suite. 3 habits keep those numbers honest. - **Look at tasks before the average.** The suite can improve overall while one important task falls from 3 of 3 to 0 of 3. Ask which tasks changed before celebrating the headline. - **Expect noise on small suites.** 30 tasks run 3 times each is 90 attempts. A move from 60 passes to 63 is within what chance alone produces at that size. Treat it as no change until more attempts confirm it. - **Ask why each failure failed.** Some failures are the agent. Some are a broken task, or a grader that is too strict. Only the first kind is a product problem. One more question matters for anything customer-facing: does it work every time? An agent that succeeds on 2 of 3 attempts sounds decent. But a person gets one attempt. Across 3 people in a row, the chance that all 3 get a good result is 2/3 × 2/3 × 2/3, about 30%. Engineers call this pass^k. Product teams can call it consistency. ## Offline evals and online evals An offline eval runs a fixed set of tasks before release. An online eval runs graders on real conversations after release. Same idea, different job. Offline tasks come in 2 kinds. **Capability evals** hold tasks the agent cannot do reliably yet, so the score starts low and gives the team something to climb. **Regression evals** hold tasks the agent already handles, and they should stay near 100%. When the skipped-contacts bug is fixed, the contact-update task moves from the first kind to the second, so the bug cannot quietly return. ## What an eval suite cannot see An eval only grades the tasks someone wrote down. People bring other tasks, and the same task in other words. Anthropic’s own guide to agent evals says production monitoring catches issues that synthetic evals miss, and that a suite which does not match real usage can create false confidence. Picture the contact-update task after the fix ships. The test list updates cleanly, so the eval passes. Then 2,400 people ask for the same thing with their own lists. 1,536 of them finish, 64%. 528 stop on **update them**. One of them writes at turn 9: “Where are the rest? Can you try that again?” No eval saw that turn, because no task contained it. The questions a product team cares most about live in those conversations, not in the suite: - **What are people asking for?** The tasks and intents they actually bring, including the ones nobody planned. - **What is working?** Which tasks finish, and whether new users get their first task done. - **What is breaking, and for whom?** The step that loses people, and the group it hits hardest. Enterprise EMEA users finish this task 52% of the time, against 64% overall. [Abracadabra](https://tryabracadabra.com/product/) answers those questions from the conversations your agent already has. It sorts each one by topic, task and intent, marks the turns where things went wrong, and lets you compare groups of users. It is error analysis at the scale of every conversation, and each failure it finds is a ready-made eval task: the request, the outcome and the turn where it broke. Explore the eval rubric library for ready-to-run binary criteria, see how the workspace works on the [features page](https://tryabracadabra.com/features/), and check [pricing](https://tryabracadabra.com/pricing/) for the free plan. For the measures that sit beside an eval suite, read [how to measure AI agent performance](https://tryabracadabra.com/blog/how-to-measure-ai-agent-performance/) and [AI agent analytics](https://tryabracadabra.com/blog/ai-agent-analytics/). **Find the tasks your eval suite does not cover.** Connect your agent’s conversations. Abracadabra shows what people ask, which tasks fail, and the turns that become your next evals. [Request access](https://tryabracadabra.com/request-access/) · [Open the playground](https://playground.tryabracadabra.com) ## Mistakes that make eval scores lie Most misleading eval results come from the suite, not the agent. These are the common causes. - **Grading what the agent said.** The text match above passed an attempt that updated 3 of 9 contacts. - **Rules 2 people would argue about.** The score then measures the grader’s guess, not the agent. - **Generic scores.** Off-the-shelf measures such as “helpfulness” sound useful and rarely match what your users need. Husain and Shankar warn they create false confidence. - **1 attempt per task.** Agents vary. A single run turns luck into a verdict. - **Only the happy path.** Test missing files, requests the agent should decline, and moments it should ask before acting. - **A suite nobody refreshes.** Tasks written from old requests drift away from what people ask now. - **Scores without conversations.** A score says something failed. Only the conversation says why. ## Eval frameworks your engineers may use A product team does not need to pick the tool, but it helps to recognise the names. Each one runs attempts, stores transcripts and compares versions. The tool matters less than the tasks in it. A suite built from real failures in any of these beats a large generic suite in all of them. ## When an eval is the wrong tool Evals are worth the effort for most agents. A few situations call for something else first. - **Nobody can say what good looks like yet.** Read 30 to 100 conversations first. Write evals once the failure types are clear. - **The output is a matter of taste.** People disagree about tone and style. Agree a rubric with examples before automating it. - **You want to know how real users fare.** An eval reports on tasks you chose. Production conversations report on the ones people chose. - **The product changes shape every week.** Tasks and rules need upkeep. An early prototype may move faster than a suite can follow. ## AI evals FAQ ### What is an eval in AI? An eval is a test for an AI system. It gives the system an input, grades what it produces, and records whether it succeeded. Teams run evals on every change to catch problems before users do. ### What is an eval in product management? For a product manager, an eval is a written definition of what good looks like, checked automatically. It turns a requirement such as “update these contacts from the trade-show list” into a task and a pass rule, so quality can be tracked from one release to the next. ### Who should write evals, product or engineering? Both, with different jobs. Product people decide which tasks matter and what counts as success, usually by reading real conversations. Engineers turn those decisions into tests and keep them running. ### What are online evals? Online evals run graders on real conversations after release, usually on a sample. They catch failures that no offline task anticipated, but they rarely have a known right answer to grade against. ### What is the difference between evals and benchmarks? A benchmark is a public test used to compare models in general. An eval is your own test of your own product, on the tasks your users bring. Benchmarks say which model is stronger overall. Evals say whether your agent does your job. ### What is LLM-as-a-judge? LLM-as-a-judge means using a language model as the grader. It reads an attempt and answers questions from a rubric, such as whether the reply said what was missing. It should be checked against human grades before you trust it. ### How many conversations should I review before writing evals? Start with at least 30, and aim for 100. Keep reading until new conversations stop showing new kinds of failure. Then write evals for the failure types that come up most. ### Do evals replace production monitoring? No. Evals test the tasks you wrote before release. Production monitoring shows the tasks people actually bring, including the ones no eval covers. Abracadabra reads production conversations by task and intent, and each failure it finds can become a new eval. ## Related guides - [**How to measure AI agent performance in production**](https://tryabracadabra.com/blog/how-to-measure-ai-agent-performance/): Requests, task outcomes, friction and policy signals, with the conversation behind each count. - [**What is AI agent analytics?**](https://tryabracadabra.com/blog/ai-agent-analytics/): Reading live conversations to find what people came to do, where the agent lost them, and why traces are not enough. - [**Conversation analytics for AI agents**](https://tryabracadabra.com/blog/conversation-analytics-for-ai-agents/): How to read what people ask an agent, and find the turn where a task failed. - [**What is AI agent observability?**](https://tryabracadabra.com/blog/ai-agent-observability/): How traces, evaluations and conversation analytics fit together. - [**Claude Managed Agents: what they are and how to run one**](https://tryabracadabra.com/blog/claude-managed-agents/): The agent, environment, session and events model, with working API calls. Sources: Anthropic, [Demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents); Hamel Husain and Shreya Shankar, [AI evals: everything you need to know](https://hamel.dev/blog/posts/evals-faq/); the [OpenAI Evals](https://github.com/openai/evals) framework.