RAG in Production: What Product Teams Get Wrong

Every RAG demo works. You point a model at a folder of documents, ask it a question, and it answers with a quote pulled straight from your data. It feels like magic, and it takes an afternoon. That afternoon is exactly why so many teams underestimate what comes next, because the demo and the product are not the same thing and the gap between them is where projects quietly die.

Retrieval-augmented generation, RAG, is the pattern of fetching relevant chunks of your own data at query time and feeding them to a model so its answer is grounded in your world instead of its training data. Teams reach for it for three good reasons: to ground answers in information the model never saw, to cut hallucination by giving the model real source text to work from, and to stay fresh without retraining anything, because you update the index, not the model. All three are real. None of them survive a naive implementation.

I have built RAG systems that held up under real traffic and I have watched clean-looking pipelines fall apart the week after launch. The failures are almost never about the model. They are about retrieval, evaluation, and the unglamorous plumbing nobody wants to own. This is what actually breaks and what to do about it.

Why the naive pipeline disappoints

The default RAG recipe you find in every tutorial goes like this: chunk all your documents into fixed-size pieces, embed each chunk into a vector, store the vectors, and at query time embed the user’s question, grab the top-k nearest chunks, stuff them into the prompt, and let the model answer. It is a reasonable starting point and it is why the demo works. It is also why the product disappoints.

The problem is that every step in that pipeline made a silent assumption, and real data violates all of them. Fixed-size chunking assumes your documents split cleanly on character count, which they do not, so you end up cutting a table in half or splitting an answer from the question it answers. Top-k retrieval assumes the right context is always in the nearest few vectors, which is false the moment a user asks something phrased differently from how the document is written. Embedding similarity assumes semantic closeness equals relevance, and it often does not, because a chunk can be about the same topic and still be the wrong answer.

On the small, clean, curated dataset in the demo, none of these assumptions get tested. On your real corpus, with its duplicates, contradictions, outdated versions, and inconsistent formatting, all of them get tested at once. The model then does its job faithfully: it builds a confident, fluent answer on top of whatever you retrieved, and if you retrieved the wrong thing it will build a wrong answer that looks exactly as polished as a right one. That is the trap. The output quality masks the retrieval failure, so teams blame the model and reach for a bigger one when the bug was three steps upstream.

Retrieval quality is the bottleneck, not the model

This is the single idea I would put on the wall. In production RAG, the model is almost never your limiting factor. Retrieval is. If the right context reaches the model, most decent models will produce a good answer. If it does not, no model on the market will save you, because you cannot reason your way to a fact that was never in the prompt.

So the work is in retrieval, and it is deeper than “pick an embedding model.” Chunking strategy is the first lever: chunk on semantic boundaries, sections, paragraphs, logical units, rather than blind character counts, and keep enough overlap that an idea split across a boundary is still recoverable. The embedding model itself matters, and the right choice depends on your domain, because a general-purpose embedding can be blind to distinctions that are obvious to a specialist reader. Metadata is the lever most teams skip and later wish they had not: tag every chunk with its source, date, section, and access scope, so you can filter before you rank instead of hoping the vector space sorts it out.

Then there is the retrieval logic itself. Query rewriting takes the user’s messy question and reshapes it into something closer to how the documents are actually written, which alone can rescue a large share of failed retrievals. Re-ranking takes the top twenty or fifty candidates from the first pass and runs a stronger, slower model over them to reorder by true relevance, so the best chunk is actually at the top rather than merely in the pile. And hybrid search, combining keyword matching with vector similarity, catches the cases each misses on its own, because vectors are great at meaning and terrible at exact terms like a product SKU or an error code, while keyword search is the reverse. Most robust production systems I have seen are hybrid plus re-ranking, not pure vector top-k. This whole layer is the part that separates a retrieval demo from retrieval that works, and it sits inside the broader question of how the pieces fit together, which I cover in LLM app architecture.

You have to measure retrieval and answers separately

Here is the mistake that keeps teams stuck: they evaluate RAG by reading the final answer and deciding if it looks good. That tells you almost nothing about why, because a good answer can come from bad retrieval by luck and a bad answer can come from perfect retrieval that the model then fumbled. If you only measure the end, you cannot tell which half is broken, so every fix is a guess.

You have to measure the two stages separately. For retrieval, the question is: did the right context make it into the prompt at all? You build a set of questions with the known correct source chunks, then measure whether retrieval actually surfaced them, using recall (did we get the right chunk in the set we retrieved) and precision (how much of what we retrieved was junk). This is answerable without ever calling the generation model, and it isolates the retrieval layer cleanly. For answer quality, given the context was correct, the question is whether the model used it faithfully, stayed grounded, and avoided inventing anything the sources did not support.

Splitting these two lets you debug like an engineer instead of a fortune teller. Low retrieval recall means fix chunking, embeddings, or search. Good retrieval but bad answers means fix the prompt or the model. Without the split you thrash. This is not a side quest, it is the core discipline of shipping AI that holds up, and it is the same muscle I lean on across every AI-native product, which is why I wrote the full method in LLM evaluation for products. Build the eval set before you scale, not after the complaints arrive.

Teach the system to say “I do not know”

The most dangerous default in RAG is that the model will always answer. If retrieval returns nothing relevant, the naive pipeline still stuffs whatever it found into the prompt and the model still produces a confident, fluent, wrong answer, because that is what these models do when handed thin context and asked to help. In a demo you never see this. In production, on the long tail of questions your corpus does not actually cover, it happens constantly, and each instance is a small erosion of the trust the whole product runs on.

The fix is to make “I do not know” a first-class, designed outcome rather than a failure. That means setting a relevance threshold on retrieval, so that when the best chunk is not good enough you treat it as no-context rather than weak-context. It means instructing the model explicitly that if the provided sources do not contain the answer, it should say so plainly instead of guessing. And it means designing the interface so that an honest “I could not find this in your documents” reads as the product being trustworthy, not broken.

Users forgive a system that admits its limits. They do not forgive one that confidently makes things up, because a confident wrong answer is worse than no answer: it costs them the time to act on it and the trust they will not give back. Grounding the model in your data reduces hallucination, but it never reaches zero, so the product has to assume a wrong answer will occasionally happen and contain the cost when it does. That containment mindset is the throughline of shipping AI features safely, and it applies to RAG as sharply as to anything.

Freshness, indexing, and the data pipeline nobody wants to own

The pitch for RAG includes freshness: update your index, not your model, and the system knows the new thing immediately. That is true, and it is also where a lot of production systems rot, because keeping an index fresh is a data engineering problem wearing an AI costume, and AI teams are often not staffed for it.

Your documents change. New ones arrive, old ones get edited, some get deleted, and the versions people rely on shift under you. Every one of those events has to propagate into your index or the system starts answering from a stale reality, which is arguably worse than not answering, because it is confidently and specifically out of date. So you need a real pipeline: detect what changed, re-chunk and re-embed only what needs it, add the new vectors, and remove the ones tied to deleted or superseded content so old chunks stop surfacing. Deletion is the step teams forget, and forgotten deletions are how a system keeps citing a document that no longer exists.

This is ongoing operational work, not a one-time load job, and it wants an owner and monitoring like any other production data flow. The uncomfortable truth is that the quality of your RAG product is capped by the quality of this pipeline. A brilliant retrieval and generation stack sitting on a stale, half-synced index produces confidently outdated answers, and users cannot tell the difference between “the model is wrong” and “the index is old.” To them it is all just the product being unreliable.

Latency and cost add up fast

A single model call is already slow and expensive compared to a database read. RAG adds more calls, and if you are not deliberate the latency and cost stack up until the experience gets sluggish and the bill gets ugly.

Walk the request path. You embed the query, that is a call. You search the vector store, that is a lookup that grows with your corpus. If you rewrite the query, another model call. If you re-rank, another model pass over the candidates. Then the generation call itself, whose cost scales with how much retrieved context you crammed into the prompt. Each step buys you retrieval quality and each step costs milliseconds and money, and the naive instinct to retrieve more chunks to be safe makes both worse, because you are paying to send the model text it does not need and slowing every response to do it.

The discipline is to treat this as a product decision, not an afterthought to optimize once it works. Retrieve enough context to answer, not the maximum you can fit. Cache embeddings for repeated queries and cache answers where questions repeat, which they do more than you expect. Use a fast, cheap model for the easy retrievals and reserve the stronger, slower one for the hard cases, the same portfolio thinking that runs through every decision in building AI-native products. Latency and cost are not engineering trivia here, they shape what the experience can even be.

Access control is not optional

This is the failure that turns an embarrassing bug into a serious incident. RAG systems retrieve documents, and if you are not careful they retrieve documents the current user was never allowed to see. In a company knowledge base that means one employee’s question surfaces another team’s confidential file. In a multi-tenant product it means one customer’s data bleeds into another customer’s answer. Both are the kind of leak that ends up in a security review or a headline.

The naive pipeline has no concept of who is asking. It embeds everything into one shared vector space and retrieves purely on similarity, so a chunk from a restricted document is exactly as retrievable as a public one. The fix has to live in retrieval, not in a filter bolted on afterward. Tag every chunk with its access scope in metadata, and filter by the requesting user’s permissions before you rank, so restricted content is never even a candidate for their query. Do not retrieve broadly and then try to strip disallowed chunks from the prompt, because that is the pattern that leaks the day someone forgets a check.

Test this explicitly with an adversarial eye. Build cases where a user asks for something they should not have access to and confirm the system retrieves nothing it should not, and re-run those cases whenever the retrieval logic changes. Access control is easy to get right when you design for it from the first chunk and painful to retrofit after you have embedded a whole corpus without scope tags. Decide this before you index, not after the leak.

When RAG is the wrong tool

RAG has become the reflex answer for “make the model know our stuff,” and reflexes are worth questioning, because a good chunk of RAG projects would be better served by a different tool entirely. Reaching for retrieval when the problem is not a retrieval problem is how teams build a heavy pipeline to solve something simpler.

If you need the model to adopt a consistent style, format, or narrow behavior rather than recall specific facts, fine-tuning or careful prompting fits better than retrieval, because that is a behavior problem, not a knowledge-lookup problem. If the answer requires taking an action or fetching live state, checking an order status, running a calculation, hitting an API, then tool use is the right pattern and RAG is a detour, since the truth lives in a system call, not a document. And if your entire relevant corpus is small enough to fit in a modern model’s context window, you may not need retrieval at all: put the documents in the prompt and skip the whole index, because long context has quietly eaten a slice of what used to demand RAG. The honest question is not “how do we do RAG here” but “does this problem need retrieval at all,” and sometimes the strongest architecture is the one you did not build.

A pragmatic path to RAG that holds up

Putting it together, here is the sequence I would follow rather than starting from the tutorial pipeline and hoping. Start by being sure retrieval is actually the right tool for the problem in front of you, using the test above. Then build your evaluation set before you build the system, a set of real questions with known correct sources, because you cannot improve what you cannot measure and you will want the baseline from day one.

Begin with the naive pipeline as a baseline, not a destination. Measure retrieval quality against your eval set first, in isolation, and only once retrieval is genuinely surfacing the right context do you turn to answer quality. Add the retrieval upgrades in response to measured failures, not on faith: hybrid search, query rewriting, re-ranking, better chunking, whichever your numbers say you need, one change at a time so you can tell what actually helped. Design the no-context path and the access-control filtering in from the start, because both are painful to retrofit and cheap to build early. Stand up the indexing pipeline as real operational infrastructure with an owner, monitoring, and a deletion path. Then ship narrow, watch real queries, feed the failures straight back into your eval set, and improve on evidence.

That path is slower than the afternoon demo and it produces something the demo never will: a system that still works when real users ask real questions your corpus half-covers. The teams that get RAG right in production are not the ones with the cleverest pipeline. They are the ones who treated retrieval quality as the main problem, measured the two stages separately, and did the unglamorous plumbing that the demo let them skip.

The short version

• Every RAG demo works because the demo data is clean. The demo and the production product are different things, and the gap between them is where projects stall.
• The naive “embed everything, top-k, stuff the prompt” pipeline disappoints because every step makes an assumption that real, messy data violates at once.
• Retrieval quality is the bottleneck, not the model. If the right context reaches the model, most models answer well. If it does not, no model saves you.
• The retrieval levers that matter: semantic chunking, the right embeddings, metadata filtering, query rewriting, re-ranking, and hybrid keyword-plus-vector search.
• Measure retrieval and answer quality separately, or you cannot tell which half is broken and every fix is a guess.
• Make “I do not know” a designed outcome. A confident wrong answer costs more than an honest non-answer.
• Freshness is a real data pipeline with an owner, including deletion. A stale index produces confidently outdated answers.
• Latency and cost stack up across embedding, search, rewriting, re-ranking, and generation. Retrieve enough, not the maximum, and cache.
• Access control lives in retrieval. Filter by permission before you rank, tag scope in metadata, and test it adversarially, or you leak documents across users.
• Sometimes RAG is the wrong tool. Fine-tuning, tool use, or long context may fit better. Ask whether the problem needs retrieval at all.

---

I am Deepanshu Grover, a Growth Product Manager and AI builder in Paris. If your RAG demo was magic and your RAG product is not, connect on LinkedIn or get in touch.