Evals
Evals are systematic tests of AI behaviour: datasets, graders and metrics run before and after every change. Learn offline and online evals and how to build a useful suite.
Overview
Evals are systematic tests for AI systems. An eval suite has three parts: a set of tasks with known good outcomes or clear success criteria, a way to grade each answer, and metrics that sum up how well the system did. Teams run evals before releasing a change, such as a new prompt, model, tool or retrieval setting, and on samples of live traffic, so they know whether the change actually made things better.
Ordinary software tests check that the same input gives the same output. AI systems are probabilistic and open-ended: the same question can produce different, equally good answers, and a small prompt change can improve one kind of task while quietly breaking another. Evals handle this by testing many representative cases and grading them with rules, rubrics, model judges or people.
A driving test is a good comparison. The examiner does not ask the candidate whether they feel ready. They use a fixed route with known challenges, such as a roundabout, a hill start and a parallel park, and a scoring sheet with clear faults. Evals are the fixed route and the scoring sheet for an AI system. Without them, decisions rest on a few impressive demos.
What it is
| Component | Description | Example |
|---|---|---|
| Dataset | Tasks with inputs and expected results or criteria | 150 contracts with lawyer-labelled risky clauses |
| Grader | Scores each output | Exact match, code check, rubric, model judge, human |
| Metric | Summarises scores | Success rate, recall, precision, groundedness |
| Harness | Runs the system on the dataset and collects results | A script or evaluation framework |
| Baseline | Scores of the current version | Last release: 88 percent recall |
| Slices | Subsets for analysis | Short contracts, scanned contracts, non-English |
| Held-out set | Cases never used for tuning | 30 contracts kept aside |
| Cost and latency | Operational metrics tracked with quality | Mean tokens and seconds per task |
There are two main modes. Offline evals run a fixed dataset before release. Online evals grade samples of real production traffic, often from traces, to catch problems that the dataset missed.
For agents, evals often grade both the outcome, whether the task was achieved, and the trajectory: which tools were used, in what order, how many steps it took, and whether any forbidden action occurred.
Where it came from
Machine learning has always measured models on held-out test sets and shared benchmarks. Large language models made this harder, because their outputs are free text and their uses are broad. Benchmark suites such as Stanford's HELM, introduced in 2022, tried to measure models across many scenarios and metrics. In March 2023 OpenAI open-sourced its Evals framework for writing and running evaluations.
In 2023 researchers studied using strong models as graders. The paper "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" reported that a strong model judge agreed with human preferences more than 80 percent of the time, about as often as humans agreed with each other, while also documenting biases such as favouring the first answer or longer answers. This made LLM-as-judge a standard grading method.
As agents spread, benchmarks began measuring task completion in realistic environments, for example SWE-bench for resolving real software issues and τ-bench, published by Sierra in 2024, which introduced the pass^k metric: the chance that an agent succeeds on all of k repeated attempts, a measure of consistency. Teams building products learned that public benchmarks rarely match their own tasks, so custom eval suites became standard practice, supported by tools such as promptfoo, DeepEval, Ragas and Braintrust.
How it works
1. Collect tasks
Start from real usage: support tickets, user queries, documents and past failures. Include easy, typical and hard cases, edge cases and cases where the right answer is to refuse or ask a question. Label where each case came from.
2. Define success
For each case, decide what "good" means: an exact answer, a set of facts that must appear, a rubric with criteria, or a check such as "the generated code passes these tests". The clearer the definition, the more reliable the grade.
3. Choose graders
Prefer deterministic graders, such as string matching, schema validation, running tests or checking a database state, wherever possible. Use model judges for qualities that need reading, such as faithfulness to sources or tone, with a precise rubric. Use human review to calibrate judges and to grade the hardest cases.
4. Run and compare
The harness runs the system on every case, grades the outputs and computes metrics. Because outputs vary, run each case more than once when results are close. Compare against the baseline, overall and by slice.
5. Investigate and decide
Read failing cases, not just the score. A one-point gain overall can hide a ten-point loss on an important slice. Decide whether to ship, fix or roll back.
6. Grow the suite
Every bug found in production becomes a new case. Over time the suite reflects what really goes wrong, and it doubles as a regression board for quality.
Key concepts and terms
| Term | Meaning |
|---|---|
| Eval suite | The dataset, graders and metrics used together |
| Grader | The method that scores an output |
| LLM-as-judge | Using a model with a rubric to grade outputs |
| Rubric | Written criteria that define a good answer |
| Offline eval | Evaluation on a fixed dataset before release |
| Online eval | Evaluation of sampled live traffic |
| Trajectory eval | Grading the path an agent took, not just the result |
| Baseline | The reference scores to compare against |
| Slice | A subset of cases analysed separately |
| Held-out set | Cases kept aside to detect overfitting to the suite |
| pass^k | The probability that all k attempts at a task succeed |
| Judge calibration | Checking a model judge's grades against human grades |
A worked example
This is an illustrative scenario. A legal-tech team runs a contract-review agent that flags risky clauses and cites the page where each appears.
- Dataset. 150 contracts with risky clauses labelled by two lawyers; disagreements were resolved by a third. Thirty contracts are held out.
- Graders. A code check compares flagged clauses with the labels to compute recall (risky clauses found) and precision (flags that are real). A second code check confirms that each cited page contains the clause. A model judge, calibrated against lawyer grades on 40 cases, scores whether each explanation is accurate.
- Baseline. The current version scores 88 percent recall, 91 percent precision and 96 percent citation accuracy, at a mean of 41 seconds per contract.
- Change. A new model is tested. Recall rises to 93 percent and precision stays at 91 percent, but citation accuracy falls to 89 percent.
- Investigate. Reading failures shows the new model cites the page where a clause is referenced, not where it is defined.
- Fix. The team adds a citation-checking step and re-runs the suite: recall 93, precision 92, citation accuracy 97, mean 47 seconds.
- Confirm. On the held-out 30 contracts the results are similar, so the change ships. Online evals then sample 5 percent of production reviews each week.
Strengths and limitations
Strengths:
- Decisions rest on evidence rather than impressions.
- Regressions are caught before users see them.
- Teams can iterate faster with confidence.
- Shared metrics align engineers, product owners and domain experts.
- A growing suite captures lessons from every incident.
Limitations:
- Good datasets and labels take real effort and expertise.
- Datasets drift away from real usage unless refreshed.
- Model judges have biases and need calibration.
- Metrics can be gamed if optimised blindly.
- Small suites give noisy results; small differences may be chance.
How it compares with alternatives
| Approach | Role | Limitation |
|---|---|---|
| Vibe checks | Quick impressions from a few examples | Not reliable for decisions |
| Public benchmarks such as lm-evaluation-harness suites | Compare general model ability | Rarely match your own tasks |
| Unit tests | Deterministic code behaviour | Cannot judge open-ended answers |
| Custom evals | Quality of your system on your tasks | Need building and upkeep |
| A/B tests | Real user impact | Slow, and only after evals pass |
| Red teaming | Finding unsafe or adversarial failures | Exploratory rather than a fixed score |
Practical guidance and best practices
- Build the dataset from real tasks and real failures, not only invented examples.
- Write down what success means for each case before grading.
- Prefer deterministic graders; use model judges with rubrics and calibrate them against people.
- Track cost, latency and steps alongside quality.
- Keep a held-out set that is never used for tuning prompts or settings.
- Report results by slice, and read failing cases, not just averages.
- Repeat runs when differences are small, and look at consistency across attempts.
- Run evals automatically on every significant change to prompts, models, tools or data.
- Add a case for every production bug, and refresh the dataset regularly.
Common mistakes
- Deciding on a model or prompt from three or four demo questions.
- Using the same examples to tune and to judge.
- Trusting a model judge that nobody has compared with human grades.
- Looking at one overall score and missing a broken slice.
- Grading only the final answer of an agent and ignoring dangerous steps along the way.
- Letting the eval set go stale while real usage changes.
Frequently asked questions
What are evals in AI?
They are systematic tests that run an AI system on a set of tasks and grade the results with defined criteria and metrics. They tell you how good the system is on the work that matters and whether a change made it better or worse.
What is the difference between offline and online evals?
Offline evals use a fixed dataset before release and give comparable scores across versions. Online evals grade samples of live production traffic, which reveals problems and new kinds of request that the fixed dataset does not contain.
How big should an eval set be?
Big enough to show meaningful differences on the slices you care about. Many teams start with 50 to 200 carefully chosen cases and grow the set from real failures. Larger sets are needed to detect small improvements reliably.
Can I use a model to grade another model?
Yes, this is LLM-as-judge, and it works well for qualities such as faithfulness and helpfulness when the rubric is precise. Judges have known biases, such as preferring longer answers, so compare their grades with human grades before trusting them.
How do you evaluate agents?
Grade the outcome, such as whether the ticket was resolved or the tests pass, and the trajectory, such as tools used, steps taken, cost and any forbidden actions. Run each task several times, because agents vary between attempts.
How often should evals run?
On every significant change to prompts, models, tools, retrieval or data, and on a schedule for online samples. Automating them in the release pipeline makes them a habit rather than an event.
Glossary
- Eval: a systematic test of AI behaviour.
- Dataset: the tasks and expected results used in an eval.
- Grader: the rule, model or person that scores an output.
- Rubric: written criteria for a good answer.
- Baseline: reference scores from the current version.
- Slice: a subset of cases analysed on its own.
- Held-out set: cases reserved to check for overfitting.
- Trajectory: the sequence of steps an agent took.
- Judge bias: a systematic tendency in a model grader.
- Recall and precision: shares of true items found and of flags that are correct.
Summary
Evals turn AI quality from opinion into measurement. A good suite combines real tasks, clear success criteria, reliable graders and metrics for quality, cost and speed, run both offline before release and online on live samples. Model judges, popularised by research in 2023, make open-ended grading practical when calibrated against people, and agent evals grade both outcomes and trajectories. Build from real failures, keep a held-out set, read failing cases by slice, run evals on every change and keep the suite growing.
Related
It belongs to Operations in my stack, under Agentic AI. Others in the same group: