Why AI features take five times longer than the demo suggested

Understanding why AI projects take so long starts with a reliable pattern in AI feature work, and if you have funded one you have probably lived it. The prototype takes two days and is genuinely impressive — impressive enough that it gets shown to the board. Then it takes four months to ship, and everybody is surprised and slightly suspicious.

Nothing has gone wrong. The demo simply proved the easy part, and the easy part is not where the money goes.

What the demo did not have to handle

A prototype runs on inputs its author chose, once, with no cost ceiling and no consequence for being wrong. Production removes all four of those at the same time.

The engineering that closes each of those gaps is the four months.

Where the time actually goes

Measuring quality. Before anything can improve, you need a way to tell whether it has. That means assembling realistic examples with acceptable answers and running them on every change. Teams that skip this cannot distinguish an improvement from a regression, so they oscillate — fix a complaint, break something nobody was watching, find out six weeks later from a customer.

Getting the right information to the model. If the feature works over your own documents or records, this dominates everything else. Which material gets retrieved, how it is ranked, how fresh it is. Research on long contexts is unambiguous that supplying more material is not a fix — accuracy depends on where relevant information sits in the input, and burying the right answer among twenty near-misses measurably degrades results.

Deciding what happens when it is wrong. A product decision, not a technical one, and it has to be made explicitly. Does the system validate and retry? Show sources so the user can check? Flag uncertainty? Route to a person? Each is a different amount of work and a different user experience.

Controlling cost and latency. Instrumented, budgeted, and tuned — routing simple requests to cheaper models, caching what repeats, keeping the input tight.

Handling the provider being unavailable. It happens. What does your product do?

A worked project timeline: prototype to production

Abstractions about "where the time goes" land better against a real shape, so here is a genuine AI feature timeline — an AI-assisted document search feature for a legal services product, from prototype to production launch.

Day 1–2: the prototype. A weekend project, essentially — a hosted model, a handful of sample documents, a basic retrieval setup. It works well on the five documents the engineer tested it against, and it gets shown to the leadership team, who are impressed and ask when it can ship. The honest answer, though nobody says it yet, is that this prototype has answered exactly one question: the underlying capability exists.

Week 1: building the evaluation set. Thirty real documents from the actual corpus, with questions a lawyer would genuinely ask and answers a senior associate signs off as correct. This is unglamorous, invisible work — nothing demoable comes out of it — and it is also the single highest-leverage week in the whole project, because everything that follows is measured against it.

Weeks 2–3: retrieval quality. The prototype's naive chunking strategy turns out to split contract clauses across chunk boundaries, losing context that changes the correct answer. Rebuilding chunking around document structure — clause by clause rather than by fixed token count — takes most of two weeks and moves the evaluation score from mediocre to genuinely good. This is invisible in a demo and is the difference between a feature lawyers trust and one they quietly stop using after the first wrong answer.

Week 4: failure handling and citations. The product decision that the summary must always show which source clause it drew from, so a lawyer can verify rather than simply trust. This adds meaningful engineering — tracking provenance through the retrieval pipeline — and it is the single change that determines whether the feature gets adopted, because a legal user will not act on an unsourced claim regardless of how accurate the underlying model is.

Week 5: cost and latency instrumentation. Per-request cost tracking, a ceiling set based on the pricing tier the feature will ship under, and caching for the common queries that turn out to represent a third of real usage once the evaluation set reveals the actual query distribution.

Week 6: security review. The feature ingests uploaded documents, which is new user-supplied content flowing into a model — the kind of surface the OWASP LLM Top 10 exists to cover. A review here catches a path where an uploaded document's content could influence the retrieval ranking for other users' queries in a shared index, closed before it ships.

Weeks 7–8: limited rollout. Ten users, watched closely. The evaluation set did not include a category of query — searching across multiple document types simultaneously — that turns out to be common in real use and performs worse than everything else. Two more weeks address it specifically, informed by real usage rather than by guessing at what might be missing.

Total: roughly ten weeks from prototype to a launch the team is actually confident in — five times the two days the prototype took, and the ratio the title of this piece refers to is not an exaggeration; it is close to typical for a feature of this shape.

LLM evaluation in practice: what thirty examples actually buys you

LLM evaluation sounds like it requires a large, sophisticated test harness, and the useful version is smaller and more achievable than that reputation suggests.

Thirty realistic examples, each with an input and an acceptable output, run automatically whenever the prompt, model or retrieval logic changes — that is the whole mechanism, and it converts every subsequent change from "does this feel better" into "does the score move." The value is not in the number thirty specifically — it is in having any consistent, automated measurement, because the alternative is a team eyeballing a handful of outputs and disagreeing about whether quality improved.

Growing the evaluation set over time matters more than getting it large on day one. Every real failure mode discovered in production — the multi-document-type query in the worked example above, or any comparable gap — becomes a new evaluation example, so the set becomes a growing record of every way the feature has previously gone wrong, checked automatically on every future change. A team that treats evaluation as a one-time setup task rather than a continuously growing asset loses this compounding benefit, and tends to rediscover the same failure modes repeatedly across different releases.

The cost of building this is genuinely small relative to what it prevents — a day or two to assemble the first thirty examples, and an hour here and there to add new ones as they are discovered. Weighed against the cost of shipping a regression to production because nobody could tell a change had made things worse, it is one of the cheapest insurance policies available in this kind of build.

The cost model is genuinely different

Most software has near-zero marginal cost: the thousandth user costs roughly what the first did. AI features do not. You pay per request.

The consequences are worth internalising before you commit:

• A generous free tier can become a real expense
• Heavy users cost more than light users, which your pricing may not reflect
• Costs scale with success, which is a strange position to be in
• A single prompt change can move the bill noticeably

This is why "set a per-request ceiling" belongs in the requirements next to the functional ones. A feature can be excellent and unaffordable, and that is a worse outcome than one that never shipped.

Questions that get you a real timeline

Ask these before approving the build. They are all answerable in plain English:

1. How will we measure whether the output is good, and can I see it? If the answer is "we'll look at it", the estimate is a guess.
2. What is the cost per request, and what is the ceiling?
3. What happens when it is wrong? What does the user see?
4. What happens when the provider is down?
5. What are we shipping to a small group first, and what would make us stop?

A team with real answers has thought about production. A team that has only shown you a demo has not started yet — which is fine, as long as the plan reflects it.

What a realistic plan looks like

• Week one: build the evaluation set and establish baseline quality
• Weeks two to six: build and iterate against it, with cost instrumented from the first deploy
• Weeks seven to nine: hardening — failure handling, latency, guardrails, the interface for wrongness
• Weeks ten onward: limited rollout, watching quality, cost, latency and the support queue together

The support queue is the underrated one. It surfaces failure modes your test set was never going to contain, which is precisely the category you most need to hear about.

The reframe worth taking away

The prototype was not a waste, and it was not a lie. It answered a genuinely valuable question: is this capability achievable at all?

It just did not answer the other one — can we run this for thousands of people, affordably, and handle it being wrong? That second question is the project. Budget it as such and these builds stop surprising anyone.

Planning an AI feature development timeline that does not surprise the board

Pulling the pieces of this piece together into a planning template for the next AI feature on your roadmap — an AI feature development timeline that reflects reality rather than the prototype's apparent completeness.

Budget these phases explicitly, even if the specific durations differ from the worked example above: evaluation set construction (roughly a week), core capability iteration against that evaluation set (the largest single phase, typically several weeks), a dedicated failure-handling and interface-design phase (often skipped in initial estimates entirely), cost and latency instrumentation, a security review proportional to what the feature touches, and a limited rollout period before general availability. Naming each phase explicitly in the plan — even before knowing exact durations — is what prevents the whole project from being estimated as though only the first phase existed.

The AI project cost overrun pattern this piece describes is not really about AI specifically — it is the general pattern of underestimating verification work applied to a domain where the prototype-to-production gap happens to be unusually large. Naming the phases in advance, even roughly, converts a plan that will predictably surprise everyone into one that has already accounted for where the real time goes.

This is the same estimation discipline covered generally in how long does software take to build, applied specifically to the AI feature shape where the gap between "it works in the demo" and "it is ready for production" tends to be larger than on conventional software — worth reading alongside this piece if the project in question also has ordinary, non-AI scope attached to it, which most do.

Setting expectations before the demo, not after

The single most effective intervention available is setting the five-times framing described in this piece's title before the prototype is ever shown to anyone outside the immediate team — not as a caveat that dampens enthusiasm, but as an accurate description of what a prototype has and has not yet proven. A leadership team told upfront that a two-day prototype represents perhaps a fifth of the total work reacts to a ten-week timeline as expected; a leadership team who only heard about the two days reacts to the same ten weeks as an overrun. The timeline does not change. The reaction to it does, entirely based on what was said first.

In one sentence

The prototype answers whether an AI feature is possible; the four months after it answers whether it is actually shippable — plan for both questions, not just the first one.

A practical next step

The next time an AI prototype impresses a room, resist the urge to give a ship date on the spot. Instead, commit to coming back within a week with the evaluation-set plan and a realistic phase-by-phase timeline built from the pattern in this piece. That short delay, spent doing the one piece of invisible work that actually predicts the schedule, consistently prevents the larger and more painful delay of a confident date that turns out to be wrong. See also how we scope a build for the specification format that makes this concrete.