Almost every AI pilot works. That is the confusing part, because the pilot working is what convinces everyone the hard part is behind them.
Here is the pattern. Someone builds a demo in two weeks. It handles the examples people bring to the meeting. Leadership is impressed, the budget gets approved, and then the thing spends nine months not going live. Nobody can point at the moment it failed, because it did not fail. It just never became real.
The reason is that a pilot and a production system are different pieces of software with the same name.
A pilot has to work on the examples someone chose. Production has to work on whatever arrives, including the malformed input, the customer who writes in three languages in one message, the document that is a photo of a fax. A pilot needs one person to be impressed. Production needs a team to change how they work on a Tuesday. The pilot proves the model can do the thing. That was never really in question. What is in question is everything around it.
So the pilot proved the wrong thing, and the nine months get spent discovering what was never proven. When leaders can name the work, it stops looking like drift.
The Five Things No Pilot Proves
Nobody knows if it is right. During the pilot, a human eyeballed every output and said yes, that looks good. That does not scale, and it is not a measurement. Production needs a real evaluation: a set of cases with known correct answers, run automatically, so you can say the system is right 94 percent of the time on this kind of input rather than it seemed good in the demo. Almost no pilot has this, and building it is usually the longest single item on the list. It is also the one that decides whether the project survives, because without it, every future change is a guess.
Nobody knows what it costs. Pilot volume is tiny, so cost never comes up. Then someone models real volume and the number is either fine or the project dies — and most teams find out at the end instead of the beginning. Worse, agentic systems do not have a fixed cost per request. The same request can cost a fraction of a cent or several dollars depending on how much work the system decides to do. If nobody has looked at the expensive tail, the budget is fiction.
Nobody knows what happens when it is wrong. Not if. When. Does a human see it before the customer does? Is there a confidence threshold that routes the uncertain cases to a person? What is the fallback when the model is unavailable? A pilot has no answer because in a pilot, the answer is that the person running the demo notices. Production needs that decision written down and built.
Nobody has to change their job yet. This is the one that kills more projects than the technical items, and it is the least discussed. The pilot ran beside the real process. Production means someone's actual daily workflow changes, and that person was not in the room when this was scoped. If they do not trust the output, they will check it manually, which means you have added work rather than removed it.
Nobody is watching it. Your existing monitoring does not understand this system. It can tell you the service is up. It cannot tell you the outputs got worse last week, which is what actually happens when a model version changes underneath you or your inputs drift. Production means someone sees quality move before a customer reports it.
Stop Treating the Pilot as Evidence You Are Almost Done
None of that is exotic engineering. It is the ordinary discipline of shipping software, applied to a new kind of component. But it does not fit in the two weeks that produced the demo, and it is invisible in the demo, which is why it never makes it into the plan.
The fix is not to stop building pilots. Pilots are useful. The fix is to stop treating the pilot as evidence that you are almost done, and to scope the five items above as the actual project, with the demo as week one of it.
The Exercise
Take your most promising AI pilot and ask the five questions.
How do we measure whether it is right?
What does it cost at real volume, including the worst requests?
What happens when it is wrong?
Whose job changes, and have we asked them?
Who sees it when quality drops?
If you cannot answer four of the five, the pilot is not close to production, whatever the demo looked like. That is not bad news. It means the remaining work is finally visible, and visible work can be scheduled.
Start with the first one. For most teams, that is the weakest, and everything else depends on it.