You evaluate an AI agent's reliability by measuring task completion on real cases — the percentage of genuine, messy, historical examples it carries from start to a correct end state without help — not by watching it perform in a demo. The distinction between step accuracy and task completion is where most evaluations go wrong: an agent that performs each individual step correctly 95% of the time will fail somewhere in a ten-step chain roughly two times in five, because errors compound across steps. Operations run on completed tasks, not impressive steps, so the completion rate on representative work is the number that matters — and it is knowable before production, if you do the work below.
The stakes of skipping this are not hypothetical. McKinsey's State of AI in GCC survey found output inaccuracy the top barrier to scaling AI, cited by 53% of respondents — and agents raise those stakes, because an inaccurate answer waits to be read while an inaccurate action has already happened. An agent given authority over live systems on the strength of a good demo is an incident with a start date. The evaluation discipline that replaces hope with evidence has five parts.
Task-completion rate, not step accuracy
Define, per task type, what a correct end state is — and include correct escalation in the definition. An invoice agent that routes a genuinely ambiguous case to a person with the right evidence attached has succeeded; one that guesses and posts has failed even if it happened to guess right, because the behaviour is what you are certifying, not the luck.
Then measure end to end. The unit of evaluation is the case, scored on where it ended up: completed correctly, escalated correctly, or failed. Step-level metrics — extraction accuracy, classification rates — remain useful diagnostics for locating where chains break, but they are inputs to debugging, not evidence of reliability. The number to put in front of a decision-maker is: out of a hundred real cases, how many reached the right end without a human rescuing them?
Set the bar against the alternative, honestly measured. The comparison is not perfection; it is your current process, which also has an error rate — usually unmeasured. Establishing that baseline is itself clarifying, and it sometimes reveals that the bar is lower, or the problem different, than assumed.
Build the evaluation set from real cases
An evaluation set is a collection of genuine historical cases with known correct outcomes — the exam the agent must pass, and the single most valuable artefact of the whole exercise. Vendor benchmarks and synthetic examples measure the wrong distribution; your invoices, your suppliers' formatting habits, your ERP's quirks are the distribution that matters.
Three properties make a set trustworthy. Representative: pull cases in proportion to reality, including the ugly tail — the scanned-sideways document, the bundled PDF, the supplier who invoices in the wrong currency. A set built from clean examples certifies the agent for a workload you do not have. Adjudicated: record the verified correct outcome per case, not what the team remembers; where the historical handling was itself wrong, note that too — those cases test whether the agent inherits or catches your existing errors. Sized to the tail: a few hundred cases is a workable start for one task type, but rare-and-dangerous cases deserve deliberate over-sampling, because the case that occurs twice a year and costs the most is exactly what a proportionally sampled set misses.
Expect to spend real effort here — for most teams it is the majority of the evaluation work, and it is also the asset that keeps paying: every future model swap, prompt change, and vendor comparison runs against the same exam.
Classify the failures — the taxonomy matters more than the rate
Two agents with identical completion rates can carry entirely different risk, and the difference lives in how they fail. Sort every failure into three bins.
Wrong action. The agent did something incorrect — posted a mismatched invoice, sent the chaser to the wrong supplier. Cost: the error itself plus detection plus undoing. The guardrails and approval gates described in AI agents in operations exist precisely to cap this bin's consequences.
No action. The agent stalled, looped, or escalated unnecessarily. Cost: throughput and human attention. Annoying, visible, and safe — an agent whose failures concentrate here is far more deployable than its completion rate suggests.
Hallucinated success. The agent reported the task done when it was not — claimed the reconciliation matched when it did not, marked the case complete without posting. This is the dangerous bin, because it defeats monitoring itself: the queue looks clear, and the discovery arrives weeks later at month-end close. Weight these failures far above their frequency; even a small rate may be disqualifying for autonomous operation, and any repeatable case of it demands a mechanism fix — typically independent verification of the end state rather than trusting the agent's own report.
The taxonomy sets the deployment posture: an agent failing safe (no action, correct escalation) can be granted autonomy earlier and monitored lighter; one that fails active or fails silent needs tighter gates regardless of its headline rate.
Re-run everything when anything changes
An agent is not software you tested once; it sits on a model that changes underneath you. Vendor model updates — including silent revisions within a version — can shift behaviour on cases that passed for months. Prompt edits, new tools, and integration changes do the same. Reliability is therefore not a fact you established; it is a property you keep re-verifying.
The practice is regression testing, borrowed intact from software engineering: the full evaluation set runs on every change to any part of the stack, and no change ships when completion drops or — especially — when the failure mix shifts toward wrong action or hallucinated success. A rate that holds while the taxonomy worsens is a regression, whatever the headline says. This is also the strongest practical argument for building the evaluation set as a durable, versioned asset: the exam only protects you if sitting it is cheap enough to happen every time.
Shadow mode: the final exam before authority
When the agent clears the evaluation set, it has passed the written test. Shadow mode is the practical: run it on the live workload with actions disabled — it sees real cases as they arrive, decides everything, records what it would have done — while people continue doing the actual work. Then compare, case by case, the agent's intended actions against the humans' real ones.
Shadow mode catches what curated sets cannot: the malformed input that arrives at 2 a.m., the timing dependency, the case type that did not exist last quarter. Disagreements resolve into three piles — the agent was wrong (fix before launch), the human was wrong (the agent is catching existing errors; count the value), or the process was ambiguous (a definition to sharpen — often the largest and most useful pile). Run it for weeks, not days, so the workload's natural variety appears; then let the exit be a measured decision against pre-agreed thresholds for completion rate and failure mix, made the same way every later autonomy extension should be made — on evidence. That evidence-gated path from pilot to production is the same discipline covered in taking AI from PoC to production.
Deployment does not end the evaluation; it changes its tense. Sampling completed cases, tracking human-overturn rates, and watching for drift are how the same questions get asked continuously in production — the subject of monitoring AI output in production. The through-line is one sentence long: an agent's reliability is not what it did in the demo, but what you can prove it does on your worst month's work — and proof is a thing you build, before you need it.