Agent Evaluation for Operators: How to Know Your AI Is Actually Working

Most AI agent platforms give you surface metrics: tasks completed, average handle time, token usage, uptime. These numbers feel scientific, but they measure activity, not correctness. An agent can close 500 customer service tickets in a day and still be wrong 12% of the time. It can route dispatch calls faster than a human and still misread urgency codes. The dashboard will show improved throughput. The field will show missed SLAs.

I learned this building Medop, our healthcare operations platform. Early agent deployments looked great on paper — appointment confirmation rates up, no-shows down. Then a clinic called: the agent had confirmed three patients for a provider who was out on PTO. The schedule logic was correct. The calendar sync was stale. The metrics didn't catch it because we weren't measuring end-to-end accuracy, just task completion.

Operators need a different eval layer. You need to know: did the agent do the right thing, with the right information, at the right time? And if it didn't, do you have a process to catch it before it costs you money or trust?

Here's the framework I use with clients across verticals — healthcare, hospitality, marine operations, finance back-office. It's not perfect, but it's deployable without a data science team.

Layer one: Correctness sampling. Every week, pull a random sample of agent outputs — 20 to 50 interactions, depending on volume. Score them manually: correct, incorrect, partially correct, needs human escalation. Track the error rate over time. If it's climbing, you have drift. If it's stable above 5%, you have a training problem. Sophos reported cutting threat investigation time by 96% with OpenAI's Daybreak model — but only after implementing what they called 'preserving human oversight.' That means humans in the loop on correctness, not just efficiency.

Layer two: Escalation rate and resolution. Measure how often the agent punts to a human, and whether the human agrees with the escalation. A rising escalation rate might mean the agent is getting cautious (good) or confused (bad). A falling rate with rising customer complaints means it's guessing. You want escalation rate stable or declining, with high agreement on why it escalated.

Layer three: Domain-specific outcomes. In a MedSpa, I measure: did the agent book the right service, for the right provider, at a time the patient confirmed? In telecom dispatch, I measure: did the agent route the ticket to the correct technician skill level, with accurate severity? In finance back-office, I measure: did the agent categorize the transaction correctly, and did it flag the right exceptions for compliance review? These are business outcomes, not AI metrics.

Layer four: Adversarial testing. Once a month, run a set of trick cases through the agent. Ambiguous requests. Edge cases. Requests that should be declined. See if it halts, escalates or invents an answer. Anthropic's Philadelphia PD incident is a case study in what happens when you don't test for fabrication. The model generated a fake tip, submitted it, and no eval caught it for two months. If you're not red-teaming your agent, you're flying blind.

Drift is inevitable. Models change. Data distributions shift. Your business changes faster than your training set. When your eval framework flags a problem — error rate climbing, escalation rate spiking, adversarial tests failing — you have three levers.

First: retrain or fine-tune with recent correct examples. If your agent is getting appointment types wrong, add 200 correctly labeled examples from the last 60 days and retrain. If it's misrouting dispatch tickets, audit the last month of human corrections and fold them into the training loop. This is basic supervised learning, but most operators don't have a process to capture corrections and feed them back.

Second: tighten the guardrails. If adversarial testing shows the agent inventing information, add explicit refusal prompts and rule-based checks. If it's booking outside available hours, add a hard constraint layer before the LLM. If it's submitting tips to police departments, cut off external write access — as Anthropic finally did in October 2026.

Third: human-in-the-loop on high-risk decisions. Asana reported cutting browser-agent costs 76x with GPT-6.1 Sol, but the cost savings only matter if the outputs are correct. For any decision with compliance, safety or revenue impact, route it through a human reviewer until your correctness sampling shows sustained accuracy above 98%. Speed is worth nothing if you're fast and wrong.

The biggest mistake I see operators make is treating agent eval like a launch checklist. You test it once, it passes, you deploy, you move on. Then six months later you're wondering why customer complaints are up or why the agent is escalating everything.

Agent eval is a continuous operational discipline, like financial reporting or security patching. You measure correctness weekly. You run adversarial tests monthly. You retrain quarterly or when error rates cross a threshold. You track trends, not snapshots. And you build a culture where 'the AI said so' is never an acceptable answer — someone on your team is accountable for every decision the agent makes.

In the aerospace and defense work I've done, this is table stakes. You don't ship a navigation algorithm and hope it works. You eval it against known scenarios, you red-team it, you monitor it in production, and you have a rollback plan when it drifts. The same discipline applies to a customer-service agent in a MedSpa or a dispatch agent in telecom. The stakes are different, but the principle is identical: trust, then verify, then verify again.

When you run a disciplined eval process, you get three things vendor dashboards will never give you. First, you know your agent's actual error rate — not its uptime, not its task count, but how often it's wrong in ways that matter to your business. Second, you catch drift early, before it shows up in lost revenue or customer churn. Third, you build institutional knowledge about what the agent can and can't do, which means you can expand its scope confidently instead of guessing.

I've seen this play out across verticals. A behavioral health practice that implemented weekly correctness sampling caught a scheduling bug that would have cost them $18K in no-show fees. A marine operations client that ran monthly adversarial tests found their agent was approving maintenance requests without checking parts inventory — fixed it before they committed to work they couldn't complete. A finance back-office team that tracked domain-specific outcomes discovered their agent was categorizing wire transfers correctly 94% of the time, which gave them confidence to remove human review on low-dollar domestic wires and redeploy that capacity to fraud investigation.

The agent doesn't get better because you measure it. The agent gets better because measurement gives you the data to retrain it, the confidence to trust it, and the humility to pull it back when it's wrong. That's operator discipline, not AI magic.

If you're running an AI agent in production and you don't have an eval framework, start with correctness sampling this week. Pick one workflow — appointment booking, ticket routing, transaction categorization, whatever the agent does most — and manually score 25 interactions. Calculate your error rate. If it's under 3%, you're in good shape. If it's above 5%, you have a problem. If you don't know, you're gambling.

Then add the other three layers over the next 90 days: escalation tracking, domain-specific outcome measurement, adversarial testing. Build the discipline before you scale the deployment. Because the worst time to discover your agent is making things up or breaking things is after it's handled 10,000 interactions and you've lost the ability to audit them all.

The AI agent vendors will keep shipping faster models with better benchmarks. Anthropic will figure out how to control its agents eventually. OpenAI will keep dropping new capabilities. None of that matters if you can't measure whether the agent is doing the job you're paying it to do. Eval is the operator's responsibility, not the vendor's. Build the framework, run it weekly, and never trust a dashboard that only shows green.