Evals before autonomy: how to know your agent is ready

Every agent project reaches the same dangerous moment. The agent has handled everything thrown at it in the demo — all five cases — and someone says the natural next thing: “This works. Can we stop reviewing every output?” The honest answer is that nobody in the room knows, because five hand-picked cases are not a track record. Before an agent acts unsupervised, it should clear the same bar as a new hire before you hand over the keys: measured performance on real work, reviewed by someone accountable.

The golden set: real cases with known answers

Strip away the research-lab framing and an eval is a simple object: historical cases where the correct outcome is already known, run through the agent, scored. Building one is cheaper than people assume. Pull 50–100 past tickets, invoices, refund requests, or lead replies out of your actual history — real ones, not sanitized composites. For each, record what your team actually did: the category assigned, the reply sent, the approve-or-deny, the dollar amount. That answer key costs one person one afternoon, plus the mild humility of accepting your own history as the standard.

Then run the agent against all of them cold — the same tools and context it will have in production, no hints in the prompt. Score each case as match or mismatch against the human decision. The first number is usually deflating; 70–85% agreement is a normal opening result. That deflating number is worth more than any demo, because it’s the same number production would eventually have revealed — discovered here with no customers in the room.

Read every mismatch — some of them are yours

The score is the headline; the mismatches are the actual product. Read every one individually. They sort into three piles:

• The agent was wrong. Missing context, an ambiguous policy, a tool that returns too little to decide with. These become prompt and tool fixes, and the fix is usually obvious once you’ve seen three examples of the same miss.
• The human was wrong. This pile is bigger than anyone expects. The agent denied the refund the written policy says to deny; the human approved it because it was 4:50 on a Friday. Count these as agent wins and correct the answer key — honestly, even when it stings. An eval that treats every historical decision as gospel trains the agent to reproduce your team’s worst days.
• Both defensible. Judgment calls with two acceptable answers. Either loosen the scoring to accept both, or tighten the written policy so humans and agents stop guessing. The eval just found a policy gap, which is a finding in its own right.

Promotion thresholds scale with blast radius

“Ready for autonomy” is not one bar. It depends on what a wrong action costs to undo:

• Drafting — internal summaries, suggested replies a human still sends. Cheapest failure mode there is. Eyeball a sample of 20–30 outputs; if they’re consistently usable, ship it and spot-check weekly. A formal harness here is over-engineering.
• Customer-visible actions — auto-sent replies, ticket routing, published content. Demand high agreement on the golden set — we look for 95%-plus within the categories being automated — plus a standing rule that anything the agent is unsure about, or outside those categories, routes to a human instead of a guess.
• Money and irreversible writes — refunds, deletions, anything contractual. The approval gate stays even after the eval is green. What a strong eval earns is the size of the auto-approvable tier — say, refunds under $50 on case types the agent has proven — while a human keeps the pen on everything above the line.

Every change resets the clock

The golden set isn’t a graduation certificate; it’s a regression suite. Re-run it before any prompt change, tool change, or model upgrade ships — especially the upgrade, because the model that got smarter also got different, and different is precisely the property you haven’t tested. We’ve watched an upgrade raise overall agreement by two points while quietly flipping behavior on one refund category; the aggregate score hid it, and the per-case diff exposed it in a single screen. Without the eval, that lesson arrives by way of a customer. Keep the run cheap and boring — a script, a case-by-case results table, a diff against the last run. If it takes more than ten minutes to execute, it will quietly stop being executed.

The honest scope check: if the workflow sees ten cases a week, skip all of this. Review every output — it’s five minutes a day and better signal than any score. Evals pay off exactly when volume makes review-everything impossible, somewhere around a few hundred cases a month; past that line, the afternoon spent on a golden set is the cheapest insurance available.

When we ship agent automation, the eval harness ships with it — the golden set, the scoring script, and promotion thresholds agreed in writing before anything runs alone. “The agent is ready” should be a number you can show the board, not a feeling someone has to defend.