What is an eval?
An eval is how a product team writes down what “good” means for an AI agent, then checks it every time the agent changes. It turns a feeling about quality into a pass or a fail.
Technically, an eval gives an AI system an input, grades what it produces, and records whether it succeeded. AI evals are the closest thing an agent has to acceptance criteria. Engineers run them. Product people decide what they test, because only someone close to users knows which jobs matter and what a good result looks like.
This guide is written for that person. It covers why evals belong to the product team and how to find what to test in real conversations. Then it turns 1 failure into an eval, compares the 3 kinds of grader, and shows how to read the results. It ends with what an eval suite cannot see.
The example is the CRM assistant used across this site. A person on a sales team asks it to update contacts from a trade-show list. There are 9 contacts. 6 of the 9 emails don’t match, so those contacts are skipped. Then it says “Done. 3 contacts updated and logged.”
Why evals belong to the product team
Every agent change is a bet: a new prompt, a new model, a new tool. Without evals, the team finds out whether the bet paid off from support tickets. With evals, it finds out before release.
Engineers and product people catch different things. Hamel Husain and Shreya Shankar teach evals to product teams, and they put it simply. Engineers catch technical issues such as tool errors. Product people catch product failures such as unmet user expectations. A tool call that returned 200 OK can still leave a person with 3 of the 9 contacts they asked for updated. Only someone thinking about the person sees that as a failure.
An eval suite answers 4 questions a product team asks all the time:
- Did this release make the agent better or worse? On the tasks people actually bring, not on a demo.
- Which jobs can the agent be trusted with? A task that passes reliably can be promoted in the product. One that does not should not be.
- Did the fixed bug stay fixed? A known failure, once fixed, becomes a test that blocks it from returning.
- Where should the next week of work go? The failing tasks, ranked by how often people bring them.
The 5 parts of an eval
Every eval tool uses the same few ideas, sometimes under different names. Learn these 5 and any eval conversation with engineers reads the same way.
| Part | What it is | In the CRM example |
|---|---|---|
| Task | 1 request, plus a rule for what counts as success | “Update these contacts from the trade-show list.” Success is 9 updated contacts. |
| Trial | 1 attempt at the task. Agents vary, so each task runs several times. | 3 attempts at the same request |
| Grader | Whatever decides pass or fail: a script, a model, or a person | A check that counts updated contacts |
| Transcript | Everything that happened in the attempt: messages, tool use, results | Turns 1 to 8 of the conversation |
| Outcome | What actually exists when the attempt ends | 3 updated contacts and 6 skipped ones |
The last 2 are the ones to keep apart. The transcript is what the agent said. The outcome is what the person got. In this example they disagree: the agent says the contacts are updated and logged, and 6 of them were skipped. A good eval grades the outcome.
Start by reading conversations, not writing tests
The most useful eval work happens before any test exists. You read real conversations and write down, in plain words, what went wrong. Practitioners call this error analysis, and Husain and Shankar call it the most important activity in evals.
It works like user research on transcripts. There are 4 steps.
- Collect a sample. Pull 100 real conversations, or at least 30 to start. Real ones, not ones the team wrote.
- Write a note on each. One plain sentence on what went wrong, from the person’s side. “Asked for 9 contacts, got 3 updated.” Note the first thing that broke, because later problems often follow from it.
- Group the notes. Notes that describe the same problem become one failure type. Name each type so the whole team uses the same words.
- Count them. The counts tell you which failures to test first.
Here is what 100 conversations about the contact update look like after grouping. They follow the same proportions as the 2,400 attempts used across this site.
| Failure type | Step where it happens | Of 100 |
|---|---|---|
| Emails didn’t match, and the agent updated the rest anyway | update them | 22 |
| Activity was not logged | log the activity | 10 |
| Agent could not find the contacts | find the contacts | 4 |
| Finished: 9 contacts updated | all 3 steps | 64 |
The picture is clear without a single test. 36 of 100 people did not get what they asked for, and most of them stopped at the same step. Skipped contacts are the first eval to write. Keep reading until new conversations stop revealing new failure types. That is the sign the list is complete for now.
One person should own the final call on what counts as a failure, ideally the person closest to users. A committee produces vague rules, and vague rules produce scores nobody trusts.
Turn a failure into an eval
An eval is a failure type written down so precisely that a script, a model or a colleague would grade it the same way you would. For the skipped-contacts failure, the task card fits in 5 lines.
| Field | The contact-update task |
|---|---|
| Request | “Update these contacts from the trade-show list.” |
| Setup | A trade-show CSV of 9 new leads, 6 with emails that differ from the CRM. When the agent asks which contacts, the answer is “Just the new leads.” |
| Pass rule | 9 contacts updated, none skipped |
| Also check | If anything is missing, the agent says so and offers to try again |
| Attempts | 3, because agents vary between runs |
Your engineers turn the card into a test. The pass rule usually becomes 1 line of code that counts updated contacts. The product decision is everything above that line: which request, which setup, and what counts as success. Browse the production agent failure and eval rubric library for the complete contact-update rubric and 26 other copy-paste binary rubrics.
Pass or fail, not 1 to 10
Write every rule as a yes-or-no question. “Rate the reply from 1 to 10” invites arguments about whether a 6 is different from a 7. “Did the agent update all 9 contacts?” has one answer. Husain and Shankar recommend binary labels for exactly this reason: they force clearer thinking and more consistent grading.
When a result is partly right, use several yes-or-no checks instead of a scale. The contact-update reply gets 3:
- Did the agent update all 9 contacts? No.
- If any were missing, did it say so? Yes, at turn 7: “6 of the 9 emails didn’t match.”
- If any were missing, did it offer to try again? No.
1 of 3. The checks also tell the team what to fix: the agent is honest about the problem, then hands over incomplete work without offering a way forward.
Test the whole conversation
Agents rarely finish in one message. The contact update needs a clarifying question at turn 2, “Just the new leads, or existing customers too?”, before any work starts. An eval of a conversation needs something to answer that question. Either a script replies with fixed answers, or a second model plays the person from a short brief. The script is predictable. The model copes with questions nobody expected.
Some rules are about how the agent works, not what it delivers. The agent must never delete or merge a contact record, whatever else it does. Rules like that deserve their own check. Most others should not dictate the route: agents often find good routes nobody planned, and an eval that insists on one path fails them for it.
Code, model or human: choosing a grader
Something has to decide pass or fail. There are 3 options, and most teams use all of them.
| Grader | Best for | Watch out for |
|---|---|---|
| Code | Facts a script can check: files exist, a number is right, a record was saved | Valid answers in an unexpected shape |
| Model | Judgement calls on open-ended replies, graded against a written rubric. Often called LLM-as-a-judge. | It can be confidently wrong, and every run consumes judge model tokens. Check it against people first. |
| People | The reference every other grader is checked against | Slow and expensive to run on every change |
The grader you pick can flip the verdict. Here is 1 attempt at the contact-update task, graded 4 ways.
The attempt
- t1 person“Update these contacts from the trade-show list.”
- t6 agent“Updating the 9 contacts now.”
- t7 agent“6 of the 9 emails didn’t match. Updating the other 3.”
- t8 agent“Done. 3 contacts updated and logged.”
Outcome: 3 of 9 contacts updated
Code grader
Does the final reply say “updated and logged”?
Pass
It grades what the agent said, and the agent said the work was done.
Code grader
Were all 9 contacts updated?
Fail · 3 of 9
It grades the CRM records. 6 of the 9 were skipped.
Model grader
All 9 updated? No. Said what is missing? Yes. Offered to try again? No.
Partial · 1 of 3
The agent was honest at turn 7, then logged 3 contacts without offering a fix.
Human grader
Did the person get what they asked for?
Fail
They asked for the whole trade-show list. 3 of 9 is not that.
Check the model grader against people
A model grader is useful because it scales. It is also a second AI, with its own blind spots. Before its scores steer a release decision, see how often it agrees with a person.
The method is simple. A person grades 40 attempts pass or fail, without seeing the model’s answers. The model grades the same 40. Then compare.
| 40 attempts | Person: pass | Person: fail |
|---|---|---|
| Model: pass | 14 | 4 |
| Model: fail | 2 | 20 |
They agree on 34 of 40, 85%. The number that matters more is the 4 in the top right: attempts the model passed and a person failed. Those are bad results that would look fine on a dashboard. Read those 4 conversations. Usually 1 rubric calibration question is too loose, and tightening it fixes most of them.
How to read eval results
An eval run ends with a score per task and 1 for the whole suite. 3 habits keep those numbers honest.
- Look at tasks before the average. The suite can improve overall while one important task falls from 3 of 3 to 0 of 3. Ask which tasks changed before celebrating the headline.
- Expect noise on small suites. 30 tasks run 3 times each is 90 attempts. A move from 60 passes to 63 is within what chance alone produces at that size. Treat it as no change until more attempts confirm it.
- Ask why each failure failed. Some failures are the agent. Some are a broken task, or a grader that is too strict. Only the first kind is a product problem.
One more question matters for anything customer-facing: does it work every time? An agent that succeeds on 2 of 3 attempts sounds decent. But a person gets one attempt. Across 3 people in a row, the chance that all 3 get a good result is 2/3 × 2/3 × 2/3, about 30%. Engineers call this pass^k. Product teams can call it consistency.
| What you see | What it usually means |
|---|---|
| Suite up, one task down | A regression hidden by the average |
| A 2 or 3 point move on a small suite | Probably chance. Add attempts before deciding. |
| Every version fails one task | A broken task or grader, rarely the agent |
| Every task passes | The suite has stopped measuring anything. Add harder tasks. |
| Scores swing between identical runs | Too few attempts, or a rubric that needs tighter questions |
Offline evals and online evals
An offline eval runs a fixed set of tasks before release. An online eval runs graders on real conversations after release. Same idea, different job.
| Offline evals | Online evals | |
|---|---|---|
| When they run | Before release, on every change | After release, on a sample of live conversations |
| What they test | Tasks the team wrote | Whatever real people ask |
| The right answer | Known in advance | Usually unknown. The grader has to judge. |
| Best at | Stopping known failures from coming back | Spotting failures nobody wrote a task for |
Offline tasks come in 2 kinds. Capability evals hold tasks the agent cannot do reliably yet, so the score starts low and gives the team something to climb. Regression evals hold tasks the agent already handles, and they should stay near 100%. When the skipped-contacts bug is fixed, the contact-update task moves from the first kind to the second, so the bug cannot quietly return.
What an eval suite cannot see
An eval only grades the tasks someone wrote down. People bring other tasks, and the same task in other words. Anthropic’s own guide to agent evals says production monitoring catches issues that synthetic evals miss, and that a suite which does not match real usage can create false confidence.
Picture the contact-update task after the fix ships. The test list updates cleanly, so the eval passes. Then 2,400 people ask for the same thing with their own lists. 1,536 of them finish, 64%. 528 stop on update them. One of them writes at turn 9: “Where are the rest? Can you try that again?” No eval saw that turn, because no task contained it.
The questions a product team cares most about live in those conversations, not in the suite:
- What are people asking for? The tasks and intents they actually bring, including the ones nobody planned.
- What is working? Which tasks finish, and whether new users get their first task done.
- What is breaking, and for whom? The step that loses people, and the group it hits hardest. Enterprise EMEA users finish this task 52% of the time, against 64% overall.
Abracadabra answers those questions from the conversations your agent already has. It sorts each one by topic, task and intent, marks the turns where things went wrong, and lets you compare groups of users. It is error analysis at the scale of every conversation, and each failure it finds is a ready-made eval task: the request, the outcome and the turn where it broke. Explore the eval rubric library for ready-to-run binary criteria, see how the workspace works on the features page, and check pricing for the free plan. For the measures that sit beside an eval suite, read how to measure AI agent performance and AI agent analytics.
Mistakes that make eval scores lie
Most misleading eval results come from the suite, not the agent. These are the common causes.
- Grading what the agent said. The text match above passed an attempt that updated 3 of 9 contacts.
- Rules 2 people would argue about. The score then measures the grader’s guess, not the agent.
- Generic scores. Off-the-shelf measures such as “helpfulness” sound useful and rarely match what your users need. Husain and Shankar warn they create false confidence.
- 1 attempt per task. Agents vary. A single run turns luck into a verdict.
- Only the happy path. Test missing files, requests the agent should decline, and moments it should ask before acting.
- A suite nobody refreshes. Tasks written from old requests drift away from what people ask now.
- Scores without conversations. A score says something failed. Only the conversation says why.
Eval frameworks your engineers may use
A product team does not need to pick the tool, but it helps to recognise the names. Each one runs attempts, stores transcripts and compares versions.
| Framework | What it does |
|---|---|
| OpenAI Evals | An open-source framework for evaluating LLMs and LLM systems, with a registry of existing evals |
| DeepEval | An open-source Python framework with ready-made metrics that runs evals like unit tests |
| promptfoo | An open-source command-line tool for testing prompts and agents, including red-teaming |
| Inspect | An open-source framework for model evaluations from the UK AI Security Institute |
| Braintrust | Offline evals, production observability and experiment tracking in one platform |
| LangSmith | Tracing, offline and online evals, and datasets, built around LangChain |
| Langfuse | Open-source tracing and evals that a team can host itself |
| Arize Phoenix | Open-source tracing and evals, with a hosted product for larger teams |
The tool matters less than the tasks in it. A suite built from real failures in any of these beats a large generic suite in all of them.
When an eval is the wrong tool
Evals are worth the effort for most agents. A few situations call for something else first.
- Nobody can say what good looks like yet. Read 30 to 100 conversations first. Write evals once the failure types are clear.
- The output is a matter of taste. People disagree about tone and style. Agree a rubric with examples before automating it.
- You want to know how real users fare. An eval reports on tasks you chose. Production conversations report on the ones people chose.
- The product changes shape every week. Tasks and rules need upkeep. An early prototype may move faster than a suite can follow.
AI evals FAQ
What is an eval in AI?
An eval is a test for an AI system. It gives the system an input, grades what it produces, and records whether it succeeded. Teams run evals on every change to catch problems before users do.
What is an eval in product management?
For a product manager, an eval is a written definition of what good looks like, checked automatically. It turns a requirement such as “update these contacts from the trade-show list” into a task and a pass rule, so quality can be tracked from one release to the next.
Who should write evals, product or engineering?
Both, with different jobs. Product people decide which tasks matter and what counts as success, usually by reading real conversations. Engineers turn those decisions into tests and keep them running.
What are online evals?
Online evals run graders on real conversations after release, usually on a sample. They catch failures that no offline task anticipated, but they rarely have a known right answer to grade against.
What is the difference between evals and benchmarks?
A benchmark is a public test used to compare models in general. An eval is your own test of your own product, on the tasks your users bring. Benchmarks say which model is stronger overall. Evals say whether your agent does your job.
What is LLM-as-a-judge?
LLM-as-a-judge means using a language model as the grader. It reads an attempt and answers questions from a rubric, such as whether the reply said what was missing. It should be checked against human grades before you trust it.
How many conversations should I review before writing evals?
Start with at least 30, and aim for 100. Keep reading until new conversations stop showing new kinds of failure. Then write evals for the failure types that come up most.
Do evals replace production monitoring?
No. Evals test the tasks you wrote before release. Production monitoring shows the tasks people actually bring, including the ones no eval covers. Abracadabra reads production conversations by task and intent, and each failure it finds can become a new eval.