The canonical RAG diagram is three boxes: a vector store, an embedding step, and a large language model. It's genuinely useful, and for a demo it's enough. The problem is that the demo diagram gets mistaken for an architecture, and an architecture has to answer questions a demo never asks.

What the diagram leaves out

Where does the data come from, and who is allowed to see it? In a real enterprise, the same question asked by two different employees should sometimes return two different answers, because they have different entitlements. Retrieval that ignores identity and access control isn't a smaller version of the solution — it's a security incident waiting for an audit.

Then there is freshness. Documents change, get retracted, get superseded. A vector index is a cache, and every cache needs an answer to invalidation. How does a corrected policy propagate? How quickly? None of this is visible in the three-box diagram, and all of it decides whether users trust the system a month after launch.

An impressive retrieval demo tells you the technique works. It tells you almost nothing about whether the system will survive contact with the enterprise.

The parts that actually take the time

Evaluation, not generation, is where production RAG lives or dies. Without a way to measure answer quality against a known set, every prompt change is a guess and every regression is invisible until a user finds it. Observability — logging what was retrieved, what was generated, and why — is what turns "the model said something wrong" into a fixable engineering problem instead of a mystery.

And around all of it sits the unglamorous enterprise plumbing: data pipelines, identity, network boundaries, cost controls on token spend, and a fallback for when retrieval returns nothing useful. These are architecture concerns, and they are why a production RAG system looks far less like the tidy three-box picture and far more like the rest of your enterprise.

The point

None of this is an argument against RAG. It's an argument for treating retrieval as one component inside an architecture that also accounts for identity, data lifecycle, evaluation, observability, cost and failure. Draw that picture, and you have something you can put in front of an enterprise. Draw the three boxes, and you have a demo.

A proof of concept is designed to answer one question: is this possible? A production system has to answer a much harder one: is this dependable, for everyone, every day, at a cost the business can justify? Those are not points on the same line. Getting from one to the other is mostly architecture work, and pretending otherwise is how timelines slip.

The environment stops being friendly

In a PoC, the data is hand-picked, the user is you, and the network has no walls. In production, the data is messy and governed, the users have wildly different permissions and intentions, and the system lives behind identity, network and compliance controls that were never part of the experiment. Each of those is an architectural decision the PoC was allowed to skip.

A prototype proves the idea can work once. An architecture makes it work reliably, safely and affordably — for everyone, repeatedly.

Non-functionals become the whole game

Latency, availability, cost-per-request, rate limits, graceful degradation when a model or retrieval step fails — in a demo these are invisible, and in production they are most of the work. A GenAI feature that is brilliant but unpredictable, or accurate but ruinously expensive at scale, is not production-ready. Deciding those trade-offs deliberately is an architecture, not a tuning exercise.

How to close the gap

The move that saves the most time is to design the target architecture early — before the PoC hardens into something everyone assumes is nearly done. Decide the identity model, the data lifecycle, the evaluation approach and the cost envelope up front, and let the proof of concept validate against that direction. The experiment then becomes the first step toward the architecture, instead of a thing that has to be quietly rebuilt once reality arrives.