August 20, 2026 · StartupQuickstart
Evals before autonomy: how to know your agent is ready
Before an agent runs unsupervised, build a golden set of 50-100 real historical cases, score agreement, read every mismatch honestly, and set promotion thresholds by blast radius.
Every agent project reaches the same dangerous moment. The agent has handled everything thrown at it in the demo, all five cases, and someone says the natural next thing: “This works. Can we stop reviewing every output?” The honest answer is that nobody in the room knows, because five hand-picked cases are not a track record. Before an agent acts unsupervised, it should clear the same bar as a new hire before you hand over the keys: measured performance on real work, reviewed by someone accountable.
The golden set: real cases with known answers
Strip away the research-lab framing and an eval is a simple object: historical cases where the correct outcome is already known, run through the agent, scored. Building one is cheaper than people assume. Pull 50-100 past tickets, invoices, refund requests or lead replies out of your actual history. Use real ones, not sanitized composites. Anthropic's guide to building evaluations says to design evals that mirror your real-world task distribution, edge cases included, and OpenAI's evaluation best practices recommend drawing on production data for the same reason. For each case, record what your team actually did: the category assigned, the reply sent, the approve-or-deny, the dollar amount. That answer key costs one person one afternoon, plus the mild humility of accepting your own history as the standard.
Then run the agent against all of them cold, with the same tools and context it will have in production and no hints in the prompt. Score each case as match or mismatch against the human decision, in code, so the run is repeatable. The first number is usually deflating. On our projects, 70-85% agreement is a normal opening result. That deflating number is worth more than any demo, because it's the same number production would eventually have revealed, discovered here with no customers in the room.
Read every mismatch, because some of them are yours
The score is the headline; the mismatches are the actual product. Read every one individually. They sort into three piles:
- The agent was wrong. Missing context, an ambiguous policy, a tool that returns too little to decide with. These become prompt and tool fixes, and the fix is usually obvious once you've seen three examples of the same miss.
- The human was wrong. This pile is bigger than anyone expects. The agent denied the refund the written policy says to deny; the human approved it because it was 4:50 on a Friday. Count these as agent wins and correct the answer key, honestly, even when it stings. An eval that treats every historical decision as gospel trains the agent to reproduce your team's worst days.
- Both defensible. Judgment calls with two acceptable answers, the kind of case Anthropic's guide describes as one where even humans would find it hard to reach consensus. Either loosen the scoring to accept both, or tighten the written policy so humans and agents stop guessing. The eval just found a policy gap, which is a finding in its own right.
Promotion thresholds scale with blast radius
“Ready for autonomy” is not one bar. Anthropic's own guidance on agents notes that autonomy brings higher costs and the potential for compounding errors, and recommends extensive testing in sandboxed environments with guardrails. How high the bar sits depends on what a wrong action costs to undo:
- Drafting. Internal summaries and suggested replies a human still sends. This is the cheapest failure mode there is. Eyeball a sample of 20-30 outputs; if they're consistently usable, ship it and spot-check weekly. A formal harness here is over-engineering.
- Customer-visible actions. Auto-sent replies, ticket routing, published content. Demand high agreement on the golden set (we look for 95% or better within the categories being automated) plus a standing rule that anything the agent is unsure about, or anything outside those categories, routes to a human instead of a guess.
- Money and irreversible writes. Refunds, deletions, anything contractual. The approval gate stays even after the eval is green. What a strong eval earns is the size of the auto-approvable tier, say refunds under $50 on case types the agent has proven, while a human keeps the pen on everything above the line.
Every change resets the clock
The golden set isn't a graduation certificate. It's a regression suite. OpenAI's guide recommends continuous evaluation that runs evals on every change, and NIST's AI Risk Management Framework treats test and evaluation as an activity across the whole AI lifecycle rather than a one-time gate. In practice that means re-running the set before any prompt change, tool change or model upgrade ships. The upgrade matters most, because the model that got smarter also got different, and different is precisely the property you haven't tested. We've watched an upgrade raise overall agreement by two points while quietly flipping behavior on one refund category; the aggregate score hid it, and the per-case diff exposed it in a single screen. Without the eval, that lesson arrives by way of a customer. Keep the run cheap and boring: a script, a case-by-case results table, a diff against the last run. If it takes more than ten minutes to execute, it will quietly stop being executed.
The honest scope check: if the workflow sees ten cases a week, skip all of this. Review every output. It's five minutes a day and better signal than any score. Evals pay off when volume makes review-everything impossible, which in our experience starts somewhere around a few hundred cases a month. Past that line, the afternoon spent on a golden set is the cheapest insurance available.
When we ship agent automation, the eval harness ships with it: the golden set, the scoring script, and promotion thresholds agreed in writing before anything runs alone. “The agent is ready” should be a number you can show the board, backed by cases anyone can open.
Sources
- Define success criteria and build evaluations, Anthropic (Claude Docs)
- Evaluation best practices, OpenAI
- Building effective agents, Anthropic
- Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1, NIST
Want systems like this built for you?
We build and run data pipelines, websites, and AI automation for startups.
