Retrieval-augmented generation is often pitched as a solved problem: embed your documents, store the vectors, retrieve the top matches, and let the model answer. In practice, teams that ship this naive version end up with an assistant that confidently cites the wrong section, or worse, hallucinates an answer the retrieved context never supported.
The first failure point is chunking. Splitting documents by a fixed character count ignores structure — a table gets sliced in half, a clause loses the heading that gave it meaning. Production systems chunk with document structure in mind: headings, sections, and tables are kept intact wherever possible, with overlap tuned per document type rather than applied uniformly.
The second failure point is retrieval quality. Pure vector similarity search misses exact-match terms — a policy number, a SKU, a defined term — that keyword search would catch instantly. Production systems need hybrid retrieval: semantic search combined with keyword or BM25 signals, re-ranked before the answer is generated.
The third, and most overlooked, failure point is the absence of grounding checks. A well-built RAG system should be able to say 'I don't know' when the retrieved context doesn't actually answer the question, rather than letting the model fill the gap with a plausible-sounding guess. That requires explicit grounding verification between the retrieved passages and the generated answer — not just a good prompt.
Get chunking, retrieval, and grounding right, and RAG stops being a demo trick and starts being infrastructure a team can actually rely on.