Evaluating an autonomous agent means grading what it changes, run after run.

An agent's output is the state of your systems.

A chatbot produces text that a person can read and judge. An agent looks something up, decides, changes a record and moves to the next step. What counts is what it leaves behind: the quotation it created, the slot it held, the email it did not send.

Anthropic's engineering team, writing in January 2026, describes agents as harder to evaluate for exactly this reason: they work over many turns, call tools and modify state. A final answer can be right after a wrong path, or wrong in a way no check on the text would ever catch.

One good run proves very little, because agents are inconsistent. The tau-bench study (τ-bench) ran tool-using agents through simulated customer conversations governed by real business rules. Its best function-calling agent, GPT-4o, completed fewer than half of the tasks. In the retail domain, its chance of succeeding on the same task eight times in a row was under 25%.

Three answers shape the design before any model is chosen: what the agent may touch, what it must never do alone, and whether its actions can be undone. The evaluation below tests each of them.

Start with one job and a written definition of done.

A general-purpose helper cannot be judged, and an agent nobody can judge will not survive its first bad week. Pick one narrow, repetitive job with a measured baseline: volume per week, time per item and how often a person gets it wrong.

Define done as a state you can check in the data. For an agent that turns an emailed enquiry into a draft quotation in ERPNext, done might read: a draft quotation exists for the right customer, with the items and quantities requested, priced from that customer's price list, and nothing has been sent to anyone.

The same document lists the agent's tools and what each may do: read a customer, read a price list, create a draft quotation. Each tool is narrow and typed. None gives the agent a raw database connection. OWASP names excessive functionality, excessive permissions and excessive autonomy as the three roots of what it calls excessive agency, and the tool list settles the first two.

A scenario set starts small and covers six kinds of case.

Start smaller than you expect. Anthropic suggests that 20 to 50 simple tasks drawn from real failures is a good start while an agent is young and each change moves the results a lot. In our experience, a few hundred deliberate cases weighted towards the edges tell you more than thousands scraped from logs.

The mix matters most. The table sets out the six kinds of case worth writing, with the quotation agent as the example.

Adversarial cases belong in every set. OWASP's Top 10 for LLM Applications puts prompt injection first, including the indirect kind that arrives inside a file or an email the agent reads. OWASP's separate list for agentic applications, published in December 2025, adds agent goal hijack, tool misuse, and identity and privilege abuse. Each entry is a ready source of test cases.

Refusals are tested as seriously as the work.

Anthropic's guidance is to test both the cases where a behaviour should occur and the cases where it should not, so the agent is not tuned in one direction only. We treat any change that makes an agent refuse less often as a regression until the evaluation set shows otherwise.

Refusal tests measure how often the model holds. They cannot make a miss harmless. OWASP says it is unclear whether fool-proof prevention of prompt injection exists. The UK National Cyber Security Centre wrote in December 2025 that it may never be totally mitigated in the way SQL injection can be, so the aim has to be reducing risk and impact.

So the actions that matter most sit outside the model. Spending money, deleting records and contacting a customer for the first time pass through a gate in ordinary code and wait for a person. OWASP's own mitigations point the same way: require a human to approve high-impact actions, and implement authorisation in the downstream system rather than relying on the model to decide. Refusal tests check that the agent asks. The gate covers the day it does not.

Grade the outcome, then read the transcripts.

Grade what the agent produced rather than the path it took, as Anthropic advises. Check the final state with deterministic code wherever you can: the record exists, the fields match, the price came from the right list, no email left the building. Code checks are cheap and exact.

Keep a model grader for what code cannot judge, such as whether a drafted reply is polite and complete, and calibrate it against cases a person has graded by hand.

Then read the transcripts. Anthropic is blunt that you will not know whether your graders work unless you read the transcripts and grades from many trials. Every test run should leave the record production will leave: the input, the plan, each tool call, each result and the final state, readable by someone who was not there.

An unattended agent is judged on pass^k.

Two measures describe repeated runs. pass@k is the chance that at least one of k attempts succeeds, and pass^k is the chance that all k succeed. A coding assistant under a developer's eye can live with pass@k. An agent acting on your records alone cannot.

The arithmetic is unforgiving. Take, as an illustration, a case the agent gets right 90% of the time, with each run independent of the others. The chance of eight successes out of eight is 0.9 to the power of 8, about 43%.

Run every case several times on every change. Report pass^k rather than a single pass, and require the must-refuse cases to pass on every run.

A shadow run puts the agent on live work with no authority.

The agent reads the real enquiries and proposes every action. For about a fortnight, a person approves, edits or rejects each proposal.

Three rates come out of it: proposals approved as they stood, proposals edited and proposals rejected. The edits and rejections are the most useful data the project will produce. Each becomes a new case in the scenario set, so the next change is tested against the agent's own mistakes.

Agree the exit criteria in writing before it starts: the volume it must cover, the approval-without-edit rate it must reach, and zero misses on must-refuse cases. Anthropic recommends extensive testing in sandboxed environments, with appropriate guardrails. A shadow run is that sandbox, fed with real inputs.

Release with the brake on.

Going live means raising limits step by step from a deliberately small allowance:

• A value cap on anything it creates.
• A rate limit per hour and per day.
• An allow-list of tools, each narrow and typed.
• A token and call budget per run, with an alert when it is reached. An agent that loops keeps spending until a limit stops it.
• A hold on spending, deletion and first contact with a customer, released only by a person.
• An audit trail of every step.

Every action that is not held can be undone, and the undo is tested before launch rather than after the first incident. Limits widen one step at a time as the evaluation set proves they can. At any point, the agent can be put back to proposing only.

Every model upgrade reruns the whole suite.

A model is the one part of a system that can get worse while nobody on your side changes anything. A provider ships a new version, behaviour shifts, and nobody notices until a customer does.

Pin the model version in configuration. When a new version appears, run the whole scenario set against it, several times per case, and compare the results with the current version, case type by case type. Refuse the upgrade if must-refuse or regression cases move the wrong way, whatever the headline benchmark says. Anthropic suggests regression suites should pass at close to 100%, so any drop is a signal. It also observes that teams with evaluations in place can assess a new model and upgrade in days rather than weeks.

The same suite runs on every other change too: a prompt, a tool, the retrieval index, the data. A regression blocks the release before anyone meets it in production. Kept this way, the evidence maps onto the Measure and Manage functions of the NIST AI Risk Management Framework, which helps when an auditor asks how you know. It is evidence for an audit and confers no certification.

The checklist before an agent goes live.

• One job, a measured baseline and a written definition of done.
• Narrow, typed tools and no raw database connection.
• A scenario set with routine, edge, must-refuse, ambiguous, adversarial and tool-failure cases.
• Every case run several times, with pass^k reported rather than a single pass.
• Graders checked against transcripts that a person has read.
• A shadow run with approval on every action, and exit criteria agreed in writing.
• Limits, budgets and an audit trail from the first day.
• Spending, deleting and first contact with a customer held outside the model.
• The suite wired into every change, including model upgrades.

We apply this method to agents we build and to agents we are asked to evaluate. An agent someone else built can be given the same scenario set, and the suite stays yours whoever maintains the system.