Why AI pilots stall: it is the operating model, not the model

There is a conversation that repeats almost verbatim across enterprises. A team built something with a language model. It worked. It was demoed to the executive committee and the reaction was genuine enthusiasm. Budget was approved. Then nothing shipped, and eighteen months later the same capability is being scoped again by a different team.

When you ask what went wrong, the answers are specific and technical: the data was messier than expected, security raised concerns, the vendor's costs did not scale, the team that built it moved on. Each explanation is true. None of them is the cause.

The pilot was never the hard part

A pilot and a production system are different categories of object. A pilot demonstrates that a capability is possible. A production system commits an organisation to operating that capability indefinitely, under load, with accountability when it misbehaves.

The skills those require barely overlap. A pilot rewards speed, cleverness and a narrow happy path. Production rewards observability, boring failure modes and the ability to explain a decision to someone who was not there. Teams that are excellent at the first are frequently not resourced for the second, and the gap is organisational rather than intellectual.

Enterprises do not have a proof-of-concept problem. They have an unusually large inventory of successful proofs of concept and no route to production for any of them.

Four things a pilot is allowed to skip

Every pilot takes shortcuts. That is legitimate — the point is to learn quickly. The failure is treating the shortcut list as a to-do for later rather than as the actual project.

• Evaluation. Pilots are judged by demonstration. Someone runs a handful of queries and the output is impressive. Production needs a scored evaluation set, a baseline, and a regression gate that fails a release. Without it you cannot tell improvement from luck, and you certainly cannot tell degradation from noise.
• Data lineage. Pilots run on an extract someone pulled manually. Production needs a pipeline with ownership, freshness guarantees and a defined behaviour when upstream breaks. This is usually the single largest hidden cost, and it is almost always discovered after the budget is set.
• Cost behaviour under load. Pilot inference spend is a rounding error. At production volume it becomes a line item that finance will ask about, and the design decisions that determine it were made months earlier by someone optimising for demo quality.
• Failure semantics. A pilot that produces a wrong answer is a curiosity. A production system that produces a wrong answer has consequences, so it needs defined behaviour for low confidence, a path to human review, and an audit trail. Retrofitting that means redesigning the workflow, not adding a check.

Why the second attempt usually fails too

The instinct after a stalled pilot is to try again with more governance. A steering committee is formed, a platform team is stood up, a framework is selected. This feels like progress and frequently produces less shipped software than the first attempt.

The reason is that the missing capability was never governance. It was production engineering applied to statistical systems — a specific discipline that sits between data engineering, MLOps and quality engineering. Most organisations do not have it, cannot hire it quickly at the seniority required, and cannot easily tell the difference between a candidate who has done it and one who can describe it.

So the second attempt adds process to a team that is still missing the same skill. Process does not close a capability gap. It just makes the gap more expensive to leave open.

What changes when you plan for production first

The alternative is not more rigour at the pilot stage — that kills the speed that makes pilots useful. It is being explicit that the pilot's job is to retire uncertainty about desirability, and that a separate, funded, differently-staffed phase makes it real.

Concretely, this means a few things are true from the beginning:

• The evaluation set exists before the model is chosen, and it is built with the people who will judge the output. If they cannot articulate what good looks like, that is the finding.
• Someone owns the data path in production, and that name is written down before the build starts.
• The cost envelope is a design constraint, stated in the same document as the accuracy target.
• There is a defined answer to "what happens when it is wrong" that does not depend on the user noticing.

A test you can run this week

Pick the AI capability closest to production in your organisation and ask four questions. Do not accept an answer that is a plan.

• What is the scored evaluation set, and what does it currently score?
• Who is on call if the data pipeline behind it fails on a Sunday?
• What does it cost per thousand requests today, and what would it cost at ten times the volume?
• What happens, mechanically, when it produces a confidently wrong answer?

If three of the four answers are aspirational, the capability is a pilot regardless of how long it has been running or what it is called internally. That is a useful thing to know early, and it is a far cheaper finding than the one that arrives when someone senior asks the same questions in front of an auditor.