Evaluation plan

Make 20 to 50 tasks from real failures. Repeat each task several times to measure pass^k (the probability that all attempts go right).
The template defines who grades and with what limits. It also sets when the agent goes to production and when it goes back.
Copy the block with the button in its corner, or open the file on GitHub.
# Evaluation plan: [agent name]

<!--
How to use this template:
- Fill it in before real users see the agent.
- Take the tasks from real failures, not from cases invented to make the agent look good.
- Run the evaluation every time the prompt, the model or a tool changes.
-->

- Date: [YYYY-MM-DD]
- Evaluation owner: [name]
- What we evaluate: model [name and version], prompt [version], tools [list and versions]

## 1. The tasks

<!--
Collect 20 to 50 tasks. Good sources: tickets that went wrong, incidents,
cases a person or the current system handles badly today.
Include cases where the right answer is to do nothing or hand off to a person.
Include prompt injection attempts (see mapa-de-amenazas.md).
-->

| ID | Input | Expected result | Origin | How it is graded |
|---|---|---|---|---|
| T01 | [...] | [expected end state] | [real failure / frequent case / edge case / attack] | [code / model / person] |
| T02 | [...] | [hand off to a person] | [...] | [...] |
| [...] | [...] | [...] | [...] | [...] |

## 2. Repetitions and metric

<!--
pass@k: at least one of k attempts succeeds. Useful when a person picks the best attempt.
pass^k: all k attempts succeed. Useful when the agent serves each customer on its own.
At 75% success per attempt, passing 3 out of 3 gives about 42%.
-->

- Attempts per task (k): [__]
- Main metric: [pass^k | pass@k], because [reason]
- We also measure: cost per successful task, steps per task, actions blocked by the gate (the control that stops actions before they run).

## 3. Who grades and how

| Grader | What for | Limits |
|---|---|---|
| Code | End state, format, which tool was called, amounts. First choice. | Measures only what you can write as a rule. |
| Model as judge | Free text, tone, whether the answer uses the right source. | On yes-or-no questions, such as whether the agent completed the task, it can agree with people as much as people agree with each other. Each model fails in its own direction: some reject successes and others approve failures. It can be consistent and still show position bias (it prefers an answer because of where it appears). Compare it with people's grades before you trust it. |
| Expert person | Calibrating the judge, ambiguous cases, reviewing samples. | Slow and expensive. Gets tired after many cases in a row. |

- Grade the result (the end state), not the path the agent took. But check the action log for side effects along the way.
- Before blaming the agent, read the transcripts of the failures. Some failures are grader bugs.
- If you use a model as judge, compare its grades with a person's on [n] tasks. Minimum agreement: [__]%. Measure it with Cohen's kappa, which discounts chance agreement. Don't measure it as exact match.

## 4. Exit criterion

The agent goes to production if it meets all of this:

- [ ] pass^[k] of at least [__]% on the task set.
- [ ] No forbidden action got through the gate.
- [ ] No injection attempt produced an external action.
- [ ] Cost per successful task of at most [__].

How it starts: a person reviews 100% for [n weeks or n cases]. After that, open it up by category: [category order].

## 5. Rollback criterion

| Signal in production | Threshold | What we do |
|---|---|---|
| Reopened cases or complaints | [__]% | [back to suggestion mode] |
| Human corrections | [__]% | [...] |
| Cost per successful task | [__] | [...] |
| Unexpected external action | 1 | Shut down and follow the runbook (the written procedure for incidents). |

Who decides to roll back: [role]. How: [switch, human queue].

## 6. Regression

<!-- Regression: a failure that was fixed and comes back. -->

- [ ] Every confirmed failure in production becomes a new task.
- [ ] Deterministic tests (with a mocked model, they always give the same result) run separately from this evaluation.
- [ ] The evaluation runs on every change to the prompt, the model or a tool.

## Why this is here

- Start with 20 to 50 tasks drawn from real failures, keep pass@k and pass^k apart, and read the transcripts: *Demystifying evals for AI agents* (Anthropic, 2026).
- Across several attempts per task, consistency drops. A computer-use agent solved about 78% of the tasks at least once in 10 attempts. It solved about 36% in all 10: *On the Reliability of Computer Use Agents* (Gonzalez-Pumariega et al., 2026).
- On whether the agent completed the task, a simple judge can agree with people as much as people agree with each other. But models fail in opposite directions: some reject successes and others approve failures: *Benchmarking LLM Judges for Mobile Agent Evaluation* (Wang et al., 2026).
- Measuring agreement as exact match inflates it. And a judge can be consistent with itself and show position bias at the same time: *Reliability without Validity* (Norman et al., 2026).