The most common reason an AI pilot stalls is not model quality. It is that nobody can say whether it works. A demo that handles ten hand-picked examples proves little, and the first real failure erases trust. Anthropic's documentation puts the order plainly: define success criteria, build ways to test against them, then improve the prompt. Evaluations come before engineering.
Criteria that can fail
Anthropic asks for criteria that are specific, measurable, achievable, and relevant. "The system should extract invoices well" fails all four. Their own examples show the shift, such as moving from "safe outputs" to a stated share of outputs flagged out of a stated number of trials.
For most enterprise work, a handful of dimensions cover it: task accuracy, consistency across rephrasing, the cost of an error, latency, and price. Anthropic also lists privacy preservation and context utilization as common criteria. Choose the three or four that decide whether the system is worth running, and give each a number you will accept.
Build the set from real work
Anthropic's advice for the eval set has three parts. Make it task-specific, so it mirrors the real distribution, including edge cases such as irrelevant input, very long input, and ambiguous cases where people would disagree. Automate grading where possible. Prefer volume over polish: more questions graded by a slightly noisier automatic grader beat fewer questions graded by hand.
We start from the last quarter's real cases, including the ones that caused rework. The ugly cases are the ones a demo skips and production does not.
Three graders
- Code. Exact match, string match, or schema checks. Use it for categories, amounts, dates, and anything with a right answer. It is fast and has no opinion.
- Another model. A rubric-based judge for tone, completeness, or whether a response contains something it should not. Anthropic recommends grading with a different model from the one that produced the output. In their multi-agent research work, a single judge call returning a score and a pass or fail was the most consistent and the closest to human judgment.
- People. Expert rubrics on a smaller sample. In the same research work, human testers caught a bias the automated checks missed: the agents favored search-optimized content farms over authoritative sources.
For systems that change state in other systems, Anthropic suggests checking the end state rather than judging every intermediate step. Agents can reach a correct result by different paths.
An example spec
This is an example for invoice extraction. The thresholds are placeholders. Set yours from the baseline you measure on your own cases.
| Criterion | Measure | Grader | Example threshold |
|---|---|---|---|
| Field accuracy | Exact match on vendor, date, total, currency | Code | Agree to a number with the AP lead |
| Arithmetic | Line items sum to the stated total | Code | No unflagged mismatches |
| Review routing | Low-confidence and rule-breaking invoices reach a person | Code on labeled set | No known-bad invoice auto-filed |
| Consistency | Same invoice, reformatted, extracts the same | Code | Agree to a number |
| Explanations | Reviewer-facing note names the problem and cites the page | Model judge, spot-checked by a person | Pass rate you accept |
| Cost and latency | Per-document cost and time | Logs | Within the business case |
Two rows are worth a comment. The review-routing row is the safety property. It does not ask the model to be right on every invoice. It asks the system to send the doubtful ones to a person. And the arithmetic row is graded by code because arithmetic is the job of code.
- Golden setreal cases, with answers
- Run systemevery change
- Code gradersexact, schema, math
- Model judgerubric, different model
- Personsamples and disputes
- Reportgates the release
The eval does not stop at launch
Run it on every change to a prompt, model, or tool, and run a sampled version on live traffic reviewed by a person. Model versions are pinned, and Anthropic publishes retirement commitments for them, so you will migrate. The eval is what lets you do that on a schedule you choose instead of finding out what changed in production.
A note on delegation
Anthropic Academy's AI Fluency course teaches a four-part framework: delegation, description, discernment, and diligence. It maps well onto this work. Delegation is what you hand to the system. Description is the prompt and the specification. Discernment is the evaluation, and diligence is the governance around it. Teams that skip discernment end up with a system nobody can defend.
This is a reference design. It describes how we would build the system and cites Anthropic's documentation for platform facts. It is not a client case study, and the figures it uses are list prices or worked examples, not results.
Have a process like this?
Bring it to a 45-minute working session.