A working AI demo takes a competent engineer about a week. A working AI feature that ten thousand customers can rely on takes a quarter. Almost every stalled AI project we are asked to rescue died in the gap between those two sentences, and the causes repeat with unusual consistency.

Failure one: nobody defined what good looks like

"It should answer support questions" is not a specification. "It should resolve 40% of tier-one tickets without escalation, with under 2% incorrect answers, measured weekly against a fixed set of 200 real tickets" is. Without the second version there is no moment at which anyone can say the thing is ready, so it never ships.

Write the target before the prompt. If you cannot state the threshold, you are not ready to build — you are ready to run a two-week evaluation.

Failure two: the data was never actually available

Pilots run on a hand-picked export. Production runs on live systems with permissions, stale records and customers who deleted their account last Tuesday. The unglamorous work of getting clean, current, access-controlled data into the model is routinely two-thirds of the project and is almost never in the pilot estimate.

Failure three: there is no fallback

Every AI feature is wrong sometimes. Features that ship have an answer to "and then what": escalate to a human, show a confidence-gated draft, refuse rather than guess. Features that do not ship are the ones where being wrong has no defined consequence, so legal and support quietly block the launch.

Failure four: evaluation has no owner

Model behaviour drifts, providers deprecate versions, your own prompt changes interact. If no one owns a regression suite that runs on every change, quality degrades invisibly until a customer complains publicly.

• A fixed evaluation set of real inputs with expected outcomes, versioned in the repository.
• An automated run on every prompt, model or retrieval change.
• A named person who reads the weekly numbers.
• A documented rollback to the previous known-good configuration.

If the feature cannot be evaluated automatically, it cannot be maintained. Everything else is a demo with a launch date attached.

None of this is exotic. It is the same discipline that separates a prototype from a product in any other domain — it just gets skipped more often when the demo is impressive enough to convince the room.