I've lost count of the AI pilots I've watched land a stunning demo and then quietly die three months later. Almost none of them failed because the model wasn't good enough. They failed in the gap between a controlled demo and a production system that runs every day, on real data, in front of real customers — and that gap is made of four specific things that nobody scoped at the start.
The uncomfortable truth is that the demo is the easy 20%. It runs on clean inputs, a friendly path, and someone watching. Production is the other 80%: the messy inputs, the unhappy path, the 3 a.m. failure with no one watching. When operators tell me their pilot stalled, I don't ask about the model. I ask about these four gaps.
Gap 1: The data was staged, not real
Every good demo runs on data that has been quietly cleaned — deduplicated records, consistent formats, the edge cases removed. Production data is the opposite: half-filled fields, three spellings of the same vendor, PDFs that are really photographs of PDFs. The agent that looked brilliant on the staged set starts guessing, and guessing in production is how you lose trust in a single afternoon.
Before I let a pilot start now, I insist on running it against a raw sample of the actual system of record — not an export someone tidied up. If the data isn't ready to feed an agent, that is the project, and no amount of model quality fixes it. The order matters: fix the data foundation first, then deploy the agent on top of it.
Gap 2: No one owns the unhappy path
A demo shows the happy path — the request the system was designed to handle. Production is defined by everything else: the input the agent can't parse, the API that times out, the case that genuinely needs a human. If there is no designed escape hatch, the agent will improvise, and an improvising agent in a real workflow is a liability, not an asset.
The fix is not more autonomy; it is a deliberate handoff. Decide, before launch, exactly which cases the agent escalates, to whom, and with what context attached. The best production systems I've shipped are confident on a narrow band and quick to hand off everything outside it. That is not a weakness in the design — it is the design.
Gap 3: You measured a demo, not an outcome
In the demo, success is 'it worked when we showed it.' In production, success has to be a number tied to the business: hours reclaimed, denial rate reduced, days-in-A/R shortened, response time cut. If you can't state the metric the pilot is supposed to move, you have no way to know whether to expand it or kill it — and pilots with no scoreboard drift until someone loses patience and cancels them.
I set the metric and the baseline before the pilot runs, not after. Ninety-five percent task accuracy sounds impressive until you realize the remaining five percent lands on a human who now has to catch it — so I also measure the cost of being wrong, not just the rate of being right. An agent that is 95% accurate but silent about its 5% is more dangerous than one that is 85% accurate and flags every case it's unsure of.
Gap 4: It was built to demo, not to be operated
The demo has no logging, no versioning, no alerting, and no one on call. Production needs all four. When an agent makes a decision in a live workflow, you need to be able to answer, later, what it did and why — for debugging, for a customer dispute, and increasingly for compliance. A system you can't audit is a system you can't trust with anything that matters.
This is the least glamorous gap and the one that most often decides whether a pilot survives. Build the boring parts — the audit trail, the version history, the alert when confidence drops — into the pilot itself. If they're an afterthought, the transition to production becomes a second project that never gets funded, and the pilot dies waiting for it.
The pattern underneath all four
Every one of these gaps has the same shape: the demo optimizes for impressiveness, and production optimizes for reliability, and those are different jobs. The teams that cross the gap successfully treat the demo as a hypothesis, not a finish line — proof that the workflow is worth automating, followed immediately by the unglamorous work of making it survivable.
So when a pilot stalls, I don't look for a better model. I look for the staged data, the missing escape hatch, the absent scoreboard, and the operability that was never built. Close those four and most pilots that were 'stuck' turn out to have been ninety percent done — they were just missing the ten percent that production actually runs on.
How to start the next one
If you're scoping a pilot now, write the four gaps down as acceptance criteria before you write anything else: the agent runs on raw production data, it has a defined handoff for everything outside its band, it moves one named metric against a baseline, and it ships with logging and alerting from day one. It is a slower start and a far higher finish rate.
The goal was never a great demo. The goal is a system that still works on a Tuesday in month six, when no one is watching and the data is as messy as it always really was. Scope for that Tuesday, and the demo takes care of itself.
