--- title: "The Demo Worked. Here's Why AI Pilots Fail in Production" description: "Gartner: at least 50% of GenAI projects died after proof of concept. Vercel's CEO named the wall: connecting agents to your systems. Here's the failure map." image: "https://storage.googleapis.com/promptmetrics-uploads/website/posts/1786796390623-752095114.jpeg" author: "Izzy" category: "Orchestration Playbooks" publishedAt: "2026-08-17T13:02:51.891Z" updatedAt: "2026-08-17T13:02:51.893Z" canonical: "https://www.promptmetrics.dev/blog/poc-easy-integration-hard" --- # The Demo Worked. Here's Why AI Pilots Fail in Production > "When we sat down and we thought about what makes it really hard to build an agent, it's actually not the proof of concept part, because anybody in the world can sit down, open Claude Code or Codex, and build an agent... the raw intelligence is already there. What's hard is securely connecting it to your systems." That's Guillermo Rauch, CEO of Vercel, on the Agent Native podcast (34:29). The host opens the episode by saying almost 1,000 people inside Vercel use the company's internal agent. So this isn't a pundit guessing. It's a CEO who runs one of the largest agent builds I know of, telling you the demo was never the test. (Quotes in this post are lightly cleaned from the auto-transcript; timestamps included so you can check me.) Your demo worked too. Then it met your CRM. In January 2026, Gartner reported that at least 50% of generative AI projects were abandoned after proof of concept ([Gartner](https://www.gartner.com/en/articles/genai-project-failure), 2026). Half. This post maps why AI pilots fail in production: five failure modes, all on the connection side, so you can name the one eating yours. > **The short version:** The proof of concept isn't the test. In January 2026, Gartner reported at least 50% of GenAI projects were abandoned after PoC ([Gartner](https://www.gartner.com/en/articles/genai-project-failure), 2026). Vercel's CEO says the hard part is securely connecting the agent to your systems. Five failure modes sit between demo and production. Here's the map, and what survives it. The PoC is easy. Connecting it to your systems is what's hard. Guillermo Rauch, CEO of Vercel, Agent Native podcast, 2026 The wall, named by someone who runs an internal agent used by almost 1,000 people. ## Why does the demo always work? In November 2025, McKinsey's State of AI survey found that 88% of organizations use AI regularly in at least one function ([McKinsey, The State of AI](https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai), 2025). Getting AI to do something stopped being the hard part years ago. That's exactly why your demo felt so good. A demo works because it removes everything production adds. Someone hand-feeds the context. Access is read-only, or there's no system access at all. There's one happy path, rehearsed. The builder is in the room, steering. No permissions, no second user, no Tuesday-afternoon edge case. Rauch's phrase for the part that's solved: "the raw intelligence is already there." > **My take:** The PoC's success is the warning sign, not the milestone. The better the demo felt, the more of the real work is still unpriced. You didn't watch the product. You watched the product minus data, permissions, writes, maintenance, and other people. The same McKinsey survey shows where the funnel narrows. In 2025, 88% of organizations used AI regularly, 39% reported EBIT impact at the enterprise level, and roughly 6% qualified as high performers ([McKinsey](https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai), 2025). Do the arithmetic: 88 minus 39 leaves a 49-point gap between using AI and profiting from it. Adoption is solved. Impact isn't. The adoption-to-impact funnel, 2025 Use AI regularly 88% Enterprise EBIT impact 39% High performers ~6% Source: McKinsey, The State of AI, November 2025 88% use AI. 39% see enterprise profit impact. ~6% are high performers. The gap is the integration work. Source: [McKinsey, The State of AI](https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai), November 2025. ## What actually breaks when AI pilots hit production? The connection breaks, not the model. In May 2025, IBM's Institute for Business Value surveyed 2,000 CEOs across 33 countries. Only 25% of their AI initiatives had delivered the ROI they expected, and only 16% had scaled enterprise-wide ([IBM IBV CEO Study](https://newsroom.ibm.com/2025-05-06-ibm-study-ceos-double-down-on-ai-while-navigating-enterprise-hurdles), 2025). The same CEOs kept raising their AI spend. They believe in the tech. Their systems don't cooperate yet. Gartner's January 2026 analysis names four reasons GenAI projects get abandoned: poor data quality, inadequate risk controls, escalating costs, and unclear business value ([Gartner](https://www.gartner.com/en/articles/genai-project-failure), 2026). Map those onto an operator's week and you get five failure modes. I'll take them one at a time. First, the itemized bill. Rauch says the hard part is "securely connecting it to your systems." What does that cost in decisions? From our own build log, one production connection, one workflow (deal-stage hygiene in a HubSpot instance, client data behind the Acme Corp placeholder convention), required: * 4 credentials issued and stored * 9 OAuth scopes granted (and 14 available scopes deliberately refused) * 23 individual permission decisions (which objects the agent may read, which fields it may propose changes to) * 3 review steps designed (draft, human approval, logged write) * 1 rollback path, tested before go-live That's one workflow. Nobody prices this in the demo. It's the whole product. One more thing before the map. Most articles on this topic headline an unverifiable "95% of pilots fail" figure circulating in vendor content; one top-ranking piece flags its own source's methodology as questionable and cites it anyway. I'm not using it. Every number here comes from a primary source with a URL and a retrieval date, listed at the bottom. If a statistic can't survive that discipline, it doesn't belong in your board deck either. ## Failure mode 1: the agent meets your real data In IBM's May 2025 CEO study, 50% of CEOs admitted the recent pace of investment left them with "disconnected, piecemeal technology" ([IBM](https://newsroom.ibm.com/2025-05-06-ibm-study-ceos-double-down-on-ai-while-navigating-enterprise-hurdles), 2025). That's the estate your agent inherits on day one. The PoC never met it. The demo saw ten clean records. Production sees six years of duplicate contacts, dead custom fields, and three tools that disagree about what a "customer" is. Gartner lists poor data quality first among its abandonment reasons for a reason ([Gartner](https://www.gartner.com/en/articles/genai-project-failure), 2026). And 68% of CEOs in the same IBM study call an integrated, enterprise-wide data architecture critical to work that crosses teams. They know the estate is the problem. Why does this kill pilots instead of just slowing them? Because the errors compound silently. An agent that summarizes a duplicate contact confidently is worse than one that fails loudly. Every wrong answer taxes the team's trust, and trust is the budget a pilot actually runs on. The fix starts with [the shape of the workflow, not the tool list](/blog/it-s-not-the-tools-it-s-the-shape): decide what "one clean record" means before the agent touches anything. ## Failure mode 2: write access without a review step Gartner names inadequate risk controls as one of its four abandonment drivers ([Gartner](https://www.gartner.com/en/articles/genai-project-failure), 2026). Here's what that abstraction looks like in your week: the pilot graduates from reading to writing, and one bad write to the CRM costs more trust than a hundred good drafts earned. The write is the moment the pilot becomes real. It's also the moment most builds skip a step. Demos are read-only. Production wants writes. Rauch's own build refuses to hand those over blind: "it doesn't just give them full read-write everything access right away. It gives you, the developer, the control" (35:05). The failure mode isn't write access itself. It's write access without draft-then-review: no way to inspect what you're approving before it lands in a production system. In Gartner's January 2026 analysis of abandoned GenAI projects, inadequate risk controls appears as a named abandonment driver ([Gartner](https://www.gartner.com/en/articles/genai-project-failure), 2026). In practice, the missing control is usually a review step: pilots that let an agent write to production systems unreviewed lose the team's trust faster than they create value. I've written up how we build this in practice: [a review step on every write to a production CRM](/blog/the-gate-belongs-inside-the-tool-giving-an-ai-agent-write-access-to-a-production-crm). Full disclosure on our own roadmap: the replayable, step-level approval object we want (an audit trail you can replay decision by decision) is designed, not yet built. The draft-then-review gate is live. The replay isn't. I'd rather tell you that than let you find out. ## Failure mode 3: the connector breaks when the tool updates The PoC is a snapshot. Production is a subscription to change. Every SaaS tool in your stack ships updates on its own schedule, and your integration inherits all of them. Rauch again, on building Vercel's internal agent: "it's actually pretty hard to assemble all of the tools, all of the frameworks, and all of the infrastructure to make something like this happen and to improve it over time" (13:24). That last clause is the failure mode. In IBM's May 2025 study, only 16% of AI initiatives had scaled enterprise-wide ([IBM](https://newsroom.ibm.com/2025-05-06-ibm-study-ceos-double-down-on-ai-while-navigating-enterprise-hurdles), 2025); "improve it over time" is where the other 84 points go to die. Field note from our own rebuild. We wired a coding agent into a real HubSpot instance (names swapped for placeholders, Acme Corp convention). The demo phase never met the things that broke first: stale properties nobody had owned since 2023, duplicate contacts with clashing lifecycle stages, and a webhook that changed shape after a tool update. Week two. The agent didn't get dumber. Its inputs moved. Demo day everything works Week 2 tool update ships, webhook changes Week 3 silent failure The PoC is a snapshot. Production is a subscription to change. Source: PromptMetrics field note, 2026 The burned-operator timeline: works for a week, breaks when a tool updates. You've probably lived the commercial version of this. The vendor's demo worked, the production build died a month after handover, and the zombie integration is still on the P&L. Maintenance is the product; the demo priced only the build. Before anyone builds against your stack again, ask the maintenance question first. I keep a list of [questions to ask before a vendor builds your integration](/blog/ai-implementation-agencies-in-the-eu-how-to-evaluate-them-2026), and "who fixes it when HubSpot ships an update" is near the top. ## Failure mode 4: the costs that only show up at scale Gartner's January 2026 analysis, drawn from hundreds of GenAI implementations, names escalating costs as a core abandonment driver ([Gartner](https://www.gartner.com/en/articles/genai-project-failure), 2026). The mechanism is boring and brutal: per-use costs that rounded to zero in the demo multiply across every user, every workflow, every day. The PoC ran fifty times. Production runs fifty thousand. Token spend scales with usage. Connector upkeep scales with the number of tools. Review time scales with the number of writes. None of that appeared in the demo, because the demo had one user, one path, and no writes. The defense is honest pricing, in both directions. When you evaluate a build, ask for the total price and the deliverable list, then ask which costs recur. Anyone quoting you an effective hourly rate or hiding the maintenance line is pricing the demo, not the production system. Remember the integration inventory above: 23 permission decisions for one workflow. Decisions cost time, and time recurs. ## Failure mode 5: nobody measured the before Gartner's fourth abandonment reason is unclear business value ([Gartner](https://www.gartner.com/en/articles/genai-project-failure), 2026). Translation: nobody timed the workflow before the pilot, so nobody can prove the pilot did anything. The project then dies in a budget meeting regardless of how well it works. I've watched this kill technically healthy builds. Baseline discipline is the cheap insurance everyone skips. Write down the scale of the task before you automate it: how many tickets, how many minutes each, how many errors per hundred. It takes an afternoon. Without it, month three becomes a budget conversation with no evidence, and "unclear value" wins by default. In September 2025, BCG surveyed 1,250 senior executives and found 60% of companies reported minimal or no material value from AI, 35% were scaling, and only 5% were "future-built," getting value at scale ([BCG](https://www.bcg.com/press/30september2025-ai-leaders-outpace-laggards-revenue-growth-cost-savings), 2025). Projects without a measured baseline can't prove which group they belong to. Who's actually getting AI value, 2025 60% minimal or no material value 35% scaling 5% future-built (value at scale) Source: BCG, survey of 1,250 senior executives, September 2025 Three tiers of AI value. Source: [BCG press release](https://www.bcg.com/press/30september2025-ai-leaders-outpace-laggards-revenue-growth-cost-savings), September 30, 2025. ## What survives production? Fewer, deeper deployments. In McKinsey's November 2025 survey, no more than 10% of organizations were scaling agents in any single business function, and nearly two-thirds hadn't begun scaling AI across the enterprise ([McKinsey](https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai), 2025). The ~6% of high performers don't have better demos. They have narrower ambitions, connected properly. That inverts the usual advice. The platform play, the everything-agent, the ten-workflow rollout: those are how you end up in Gartner's abandoned half. What survives looks smaller. Pick one workflow. Connect it properly, with the permission decisions itemized. Measure it honestly, against a baseline you wrote down first. Keep a human review step on every write. Scope is a survival trait, not a lack of ambition. I've written about [how to choose the first workflow](/blog/pick-one-boring-task); boring beats visible, every time. IBM's data shows the gap the survivors close. In 2025, 61% of CEOs said they were actively adopting AI agents, while only 25% of initiatives delivered expected ROI and 16% scaled enterprise-wide ([IBM](https://newsroom.ibm.com/2025-05-06-ibm-study-ceos-double-down-on-ai-while-navigating-enterprise-hurdles), 2025). Ambition outruns integration by 36 points (61 minus 25). And once a connected agent works, the stakes flip: Rauch says of Vercel's own build, "the whole business runs on this. Hundreds of millions of dollars of revenue are dependent on the well-being of this agent" (41:12). Production isn't the finish line. It's custody. Which raises its own question, [whose job the connection work becomes](/blog/governance-is-the-new-job), and that deserves its own post. Ambition vs. reality, 2,000 CEOs, 2025 61% 25% 16% adopting AI agents delivered expected ROI scaled enterprise-wide Source: IBM Institute for Business Value, CEO Study, May 2025 Adoption ambition vs. delivered reality. Source: [IBM IBV CEO Study](https://newsroom.ibm.com/2025-05-06-ibm-study-ceos-double-down-on-ai-while-navigating-enterprise-hurdles), May 2025. ## Frequently asked questions ### What percentage of AI projects fail after proof of concept? At least half. In January 2026, Gartner reported that at least 50% of generative AI projects were abandoned after proof of concept by the end of 2025, citing poor data quality, inadequate risk controls, escalating costs, and unclear business value ([Gartner](https://www.gartner.com/en/articles/genai-project-failure), 2026). ### Why did my AI pilot work in the demo but fail in production? Because the demo removed what production adds: live system access, messy data, permissions, writes, and other users. As Vercel CEO Guillermo Rauch put it in 2026: "it's actually not the proof of concept part... What's hard is securely connecting it to your systems" (Agent Native podcast, 34:29). ### Is the model the problem? Rarely. In May 2025, IBM found only 25% of AI initiatives delivered expected ROI across 2,000 CEOs surveyed, while the same CEOs kept increasing AI investment ([IBM](https://newsroom.ibm.com/2025-05-06-ibm-study-ceos-double-down-on-ai-while-navigating-enterprise-hurdles), 2025). The failure lives in integration, data quality, and measurement, not in raw model capability. ### How do I stop a pilot from dying at production? Baseline the workflow before you build (count tasks, minutes, and errors). Start with one workflow, not a platform. Keep a [human review step on every write](/blog/the-gate-belongs-inside-the-tool-giving-an-ai-agent-write-access-to-a-production-crm) to a production system. And budget for connector maintenance, not just the build; the tool updates are coming either way. ## Where this leaves you Three things to take into your next pilot conversation: * The PoC proves the model works. It never proves the integration works, and the integration is the product. * All five failure modes live on the connection side: real data, unreviewed writes, tool updates, scale costs, and a missing baseline. Not the model. * The survivors scope down and measure first. One workflow, itemized permissions, a written baseline, a review gate on every write. If you can name your broken workflow in one sentence, that's the entry ticket.