Why AI Projects Fail Before Production

When a company comes to us after an AI project has stalled, they almost always apologize for the wrong thing. They tell me they picked the wrong model, or moved too early, or that the technology just was not ready. Then I ask to see the demo, and the demo works. It classifies the tickets. It drafts the summary. It answers the question it was built to answer. The part everyone blames is the part that actually shipped.

That contradiction is the whole essay, so I want to sit on it. The model is usually the only piece of a failed AI project that did its job. The project died somewhere the demo never went: in the data it was forced to trust, in the systems it had to write back into, in the first moment it was wrong and nobody had decided what should happen next. This is not a hunch from client calls anymore. The industry spent two years running the experiment at enterprise scale, and the results are in.

In 2025, MIT's Project NANDA studied roughly 300 enterprise generative-AI deployments and found that about 95% delivered no measurable impact on the bottom line, even as companies spent an estimated 30 to 40 billion dollars on them. The report did not blame model quality. It blamed the fact that the systems never learned the workflow they were dropped into, and never connected deeply enough to the operation to matter. In other words, the models worked and the projects still failed.

I've spent more than a decade building enterprise systems, including stretches at JPMorgan, Morgan Stanley, and Adobe, and I ran engineering as CTO of a startup I helped build from nothing. Every AI system my team has put into production, from Trust and Safety tooling at Discord to an RFP pipeline for proposal teams, survived the part the demo skips. This piece is about what that part costs, why the receipts now prove it is the real failure point, and what it takes to build past it.

The failure is measured now, and it does not point at the model

For a long time "most AI projects fail" was a thing people said at conferences. It is now a thing several independent studies have counted, and the useful part is not the size of the numbers. It is what every one of them names as the cause.

Read those root-cause lists back to back and notice what none of them leads with. Not "the model was too weak." They lead with data quality, integration, risk controls, and a definition of value nobody wrote down. Gartner made a related point in a separate forecast, predicting organizations would abandon 60% of AI projects unsupported by AI-ready data through 2026. Every cause on those lists sits on the far side of the demo. That is the pattern worth internalizing before you spend a dollar on a model.

A demo is a lie of omission

A demo is not dishonest. It is just incomplete in a very specific, very consistent way. It shows you the one path where nothing goes wrong and stays quiet about the twenty where something does.

The input in a demo is clean. The workflow is trimmed to its happy case. The person driving does the expected thing at the expected moment. Nobody asks how the system behaves when the source document is malformed, when the permissions are wrong, when the data is three weeks stale, or when someone has to explain the output to an auditor next quarter. A strong demo proves the use case is interesting. It says nothing about whether the system is deployable, and those are different claims that get treated as the same claim in a room full of excited people.

The work that decides the project starts at the boundary between two questions. "Can the model do this once?" is a research question, and the answer is usually yes. "Can this workflow survive real usage, real data, and real accountability?" is an engineering question, and the answer is the entire project. Teams that conflate the two ship the first answer and get billed for the second.

What production actually costs

Production is not the demo in a nicer wrapper. It is everything the demo agreed not to mention, built deliberately, most of it having nothing to do with the model. When people are surprised by the engineering mix on a real AI build, this is why. The model is the headline, and the layer around it is the work. Here is what that layer is made of, with the decisions that made our builds hold up.

Evaluation you can run again tomorrow

"This looks right" is not a production metric, it is a feeling in a meeting. If a system classifies support tickets, reviews contracts, or flags risky behavior, you need a repeatable way to know whether it is getting better, drifting, or quietly getting worse, and you need it before launch, not after the first incident.

In practice that means a benchmark set with known-good answers, pass and fail criteria you can defend to a skeptic, regression checks that run when you change a prompt or a model, and monitoring that keeps scoring once the thing is live. The reason to build the harness early has no shine on it. Without it you cannot tell an improvement from a regression, so every change becomes an argument about vibes. With it, a model swap becomes a measurement instead of a leap of faith. On an admin build my team shipped, that discipline paid off in diagnosis. Because the tools, the rendering, and the model's own decisions were scored separately, when something went wrong we could tell which of the three had actually broken instead of guessing.

The data contract comes before the prompt

Most AI systems fail upstream of the model, in the data. A model cannot rescue records that are fragmented across systems, labeled inconsistently, missing the fields the decision depends on, or locked behind a permission model nobody worked through. Teams like to start with prompt design because it feels like momentum. The honest first milestone is almost always getting the data into a shape the system can trust.

That work is a contract, not a cleanup pass. You decide where each input comes from, which fields are reliable enough to act on, and what the system does with a record that is wrong or absent, and you decide it before writing a prompt. When we built more than a dozen internal Trust and Safety tools for Discord, a good share of the engineering was exactly this: wiring Snowflake and Discord's own APIs so the tools read consistent, query-able data and moved high-quality labels into the ML pipelines, all under role-based access. The moderation logic was tractable. The data foundation under it was the real build, and it cut manual overhead by 40%.

Integration is where pilots quietly die

AI produces no value in isolation. It has to live inside the systems your team already runs, which means reading from databases, respecting the auth rules those systems enforce, writing back into a CRM or an internal tool, triggering the downstream steps, and fitting into approval flows that exist for reasons predating the project. Each of those touchpoints adds surface area, and each one can break in a way a proof of concept never surfaces because the proof of concept never touched it.

The instinct on an admin integration is to build a few enormous tools that each do everything. That is a mistake, and I've watched it degrade a model's behavior in real time. When we replaced part of an admin dashboard with a ChatGPT App, the system got noticeably more reliable once we made the tools narrow, explicit, and composable, with names like getUserByEmail and getRevenueByDateRange instead of one mega-tool full of optional flags. Narrow tools are easier for a model to call correctly and easier for a human to reason about when a call goes wrong. That is not a model tuning trick. It is interface design, and it is where the environment stops fighting you.

What the system does when it is wrong

Every AI system will be wrong sometimes. That is not a risk to mitigate down to zero, it is a certainty to design around, and the useful question is what the product does in the next second. Does it fall back to a deterministic path. Does it route to a human. Does it show the user a clear error instead of a confident wrong answer. Does it log enough context that someone can reconstruct what happened three months later.

A demo can ignore all four. Production cannot ignore any of them. This is also where trust is won or lost, because the first time a system is confidently wrong with no visible seam, people stop believing the outputs they cannot see it check. On the admin app, the hardest part was never rendering data, it was permissions and recovery: a layered auth flow where the user authenticated through a one-time password tied to a registered email, the server validated a short-lived signed token on every tool call, and sensitive actions still verified role against the database, so a stale token never outlived a permission someone had already revoked. We logged every tool call and parameter set, which is what makes a bad version something you can roll back and audit rather than something you discover from an angry customer.

The human boundary is a design decision, not a disclaimer

"Human in the loop" gets said like a compliance checkbox. Done well it is an architectural line you draw on purpose: these decisions the system makes alone, these it proposes and a person confirms, these it never touches. Draw the line in the wrong place and you either bury people in rubber-stamp approvals or hand the model authority it has not earned.

When we built Vercor, an AI pipeline for RFP responses, the whole design assumed a human made the final call. We split the workflow into eight stages instead of one vague "generate the response" step, because proposal work fails when too much logic collapses into a single prompt. The last stage exists only to catch omissions, qualification mismatches, and formatting misses before a human reviewer does, so the expert spends their time on win strategy and risk, not on checking whether requirement 4.2.3(b) got answered somewhere in the draft. The system removes the drudgery. It does not pretend to remove the judgment, and drawing that boundary explicitly is what made proposal teams willing to trust it.

There is one more reason production AI looks less exciting than the pilot, and it belongs here. A meaningful amount of the reliability comes from boring infrastructure choices. We built Vercor on Cloudflare because proposal work is document-heavy, deadline-driven, and spiky across customers, and in procurement-heavy settings the infrastructure question (how documents are handled, how customer data is separated, whether an action can be traced later) is not a backend detail. It is a buyer question, and it is part of what makes the system credible.

Three questions to ask before you greenlight anything

Before committing budget, three questions surface most of the risk while it is still cheap to act on. The answers tell you whether you have a project or a wish.

When the honest answer is not AI

A good partner will sometimes tell you not to build the thing you came to build. Not every operational problem needs a model, and forcing one in is its own way of failing before production.

If the workflow is deterministic, a rules engine, a SQL query, or ordinary automation is cheaper, simpler, and more reliable than anything with a model in it, and it will not drift. AI earns its place when the input is unstructured, when the decision genuinely needs context, or when the volume makes manual review too expensive to sustain. The teams worth hiring are not the ones that reach for a model on every problem. They are the ones that can tell you, with reasons, where a model has no business being. That restraint is also the cheapest reliability you will ever buy, because the safest production AI system is the one you correctly decided not to build.

Where this leaves you

The projects that reach production are not run by the people most impressed by the demo. They are run by the people who treat the demo as the cheapest, least informative part of the work, and who spend their real effort on the data, the integration, the evaluation, and the plan for being wrong. That is the quiet discipline that turns a clever prototype into something a business can actually depend on, and it is the discipline the failure statistics are quietly measuring the absence of.

If you are choosing who builds it, what a custom AI development company actually does covers how to tell a real production shop from one that will configure a tool and call it custom, and how to choose an AI automation agency is the practical vetting checklist. If your workflow sits in the public sector, government AI solutions have to explain themselves covers the accountability an agency build has to carry that a private one does not. If the thing you are building is an agent that will take actions on its own, how to build an enterprise AI agent covers earning it that authority one reversible action at a time. If you already have a workflow in mind and want it to survive contact with real data, our AI and intelligent automation practice is built around taking projects past the demo and into production.

Frequently asked questions