Across two decades working in telecom field operations, aerospace programs, and healthcare practices, I've seen the same pattern repeat: an organization ships an AI agent, watches it run for a few weeks, and then asks me the question that matters: *Is it actually working?* Not 'Can it work in theory,' but 'Is this saving us time or money right now?' The gap between what an agent *does* and what an agent *delivers* is where most deployments fail. I'm not talking about existential AI safety — that's important at the frontier, and I follow it closely — but for SME operators, the real problem is simpler and more urgent: you need evaluation frameworks that work in production, with live data, under the constraints of a real business. This piece lays out how I approach that with my clients.
Why Standard AI Benchmarks Don't Tell You Anything
When I evaluate an agent for a PT practice, a MedSpa, or a finance operation, the first thing I don't do is run it against MMLU or HumanEval. Those benchmarks tell you how a model performs on academic tasks. They don't tell you whether your agent is actually scheduling patients faster, reducing no-shows, or cutting back-office processing time by 20 percent.
The industry has gotten better at this — organizations like Anthropic and OpenAI are now using third-party evaluators to stress-test agent behavior in more realistic scenarios. That's progress. But it's still not the same as asking: *Does this agent, in my workflow, with my data, actually reduce friction?* You need to measure against your baseline, not against a public leaderboard. The agent that gets a 92 percent accuracy score on a benchmark might still fail your most important use case — say, correctly routing complex customer inquiries to the right team member, or flagging compliance exceptions before they become audit issues.
Start with Operational Friction Points, Not Feature Checklists
I begin every agent evaluation with the same question: *What does this currently take, and what would it look like if it took half the time?* Not 'What can an AI do?' but 'What's slowing us down right now?' In a two-clinic PT practice, that might be: patient intake forms take 20 minutes per session, three staff touch the file, and half the time information is missing or inconsistent. An agent that handles intake, validates completeness, and flags missing fields before the patient leaves the waiting room has a clear, measurable ROI: fewer staff hours, fewer phone calls, faster start times.
Once you identify the friction point, define the measurement before you deploy. How many hours per week does the current process take? How many errors? How much rework? Then, after the agent runs for two weeks in production, measure the same metrics. That's your evaluation baseline. If the agent was supposed to cut intake processing time by 40 percent and it's only 15 percent, you have a real number to diagnose against — not a feeling.
Three Metrics That Actually Matter
I track three categories when I evaluate an agent in the field:
**Time saved per cycle.** Count the actual hours your team spends on the task before and after. Multiply by your fully loaded labor cost. If an agent handles customer escalation triage and cuts your support team's triage time from 8 minutes to 2 minutes per ticket, and you process 300 tickets per week, you're saving 30 hours per week. At $35/hour fully loaded, that's $1,050 per week — real money for an SME. Measure it weekly for four weeks. Variance tells you whether the agent is stable or hitting edge cases.
**Error rate and rework cost.** An agent that gets 95 percent of customer inquiry routing correct but sends 5 percent to the wrong team creates rework. Measure: How many tickets does your team have to re-route or re-handle after the agent touches them? What's the cost per error? I've watched agents that looked good on throughput actually increase your cost because every error required a senior person to fix it. Track errors by category — wrong department, missing critical info, hallucinated detail — so you can feed real failure patterns back to the team managing the agent.
**Compliance and audit readiness.** For healthcare practices, finance, or regulated industries, an agent that saves time but creates audit exposure isn't an agent — it's a liability. I measure: Does the agent's work product pass your compliance checklist? Can you audit its decisions? Is there a clear chain of custody for sensitive data? For a behavioral health practice using an agent to draft clinical notes, that might mean: 100 percent of notes are reviewed by a clinician, zero notes leave the practice, and every agent decision is logged. That's not overhead — that's the cost of doing the work safely.
Build a Phased Evaluation: Sandbox, Pilot, Production
I don't hand an operator a live agent and say 'measure it.' I structure evaluation in three phases. **Sandbox:** The agent runs on historical data or synthetic scenarios, with no real output. You measure latency, consistency, and whether it handles known edge cases. This is where you catch obvious failures without risk. **Pilot:** The agent runs on a subset of real work — say, 10 percent of incoming tickets, or one clinician's patient intake forms — with a human still making the final call. Measure all three metrics above. If error rates spike or time savings disappear in real conditions, you find out with limited exposure. **Production:** Only after pilot results justify it does the agent work at scale. Even then, I build in a monthly evaluation checkpoint: pull a sample of the agent's work, spot-check accuracy, re-measure time and error rates, and compare against the pilot baseline.
The reason I use this structure is simple: agents that look perfect on clean, historical data often behave differently when they hit the messy, varied reality of your actual workflow. A customer service agent trained on FAQ examples might reason beautifully in the sandbox, then fail when a real customer asks something slightly different. You want to catch that failure mode in the pilot, not at scale.
Watch for the Invisible Cost: Monitoring and Maintenance
One number I see operators ignore is the cost of keeping an agent working. An agent that saves you 20 hours per week but requires 8 hours per week of monitoring, prompt tuning, and error review is a net win — but a smaller win than it looks. I measure: How much time does your team spend managing this agent? What's the ratio of agent work to management overhead? For a well-built agent in a stable domain, it should be lower than 10 percent of the savings — so if you save 20 hours, you spend less than 2 managing it. If it's higher, either the agent needs better training data, the domain is too variable for current agent maturity, or the evaluation itself is catching real brittleness.
I also track: If the agent makes a critical mistake, how quickly do you catch it and how hard is it to fix? An agent that fails silently — produces plausible-looking output that's actually wrong — is worse than an agent that fails loudly. You want agents that either work reliably or clearly flag uncertainty. That's harder to measure than accuracy, but it's worth building into your evaluation: spot-check the agent's confident answers; the ones where it seems most certain are often the ones worth verifying most closely.
The One Question That Separates Working Agents from Experiments
After four weeks of measurement, I ask the operator one final question: *Would you pay for this agent separately, as a line item in your budget, if the vendor charged you its cost-per-use?* If the answer is no, the agent isn't working — not because it's a bad technology, but because it hasn't yet solved the problem in a way that justifies its existence in your particular workflow. That's valuable data. Either the agent needs retraining, the use case was wrong from the start, or the friction point you picked doesn't matter as much as you thought.
If the answer is yes, you have a working agent. Measure it monthly, adapt as your volume changes, and use the framework to decide what to automate next. I've shipped agents into SME practices that cut admin overhead by 15–30 percent. The ones that worked were the ones where the operator had a clear baseline, owned the measurement, and was willing to adjust rather than declare victory after week one. That's not magic — it's discipline.
