RAG That Answers Correctly: Retrieval Is the Hard Part
Most RAG failures are retrieval failures. How chunking, hybrid search and honest 'I don't know' behaviour make knowledge systems trustworthy.
NexusNao TeamEngineering
When RAG is wrong, look at retrieval first
Teams blame the model when their knowledge assistant answers incorrectly, but in our experience the passage the model needed usually never reached it. The generation step can't cite what retrieval didn't find.
Instrument retrieval separately from generation. Measure whether the right chunks appear in the context window at all — that single metric explains most quality problems.
Chunking is a modelling decision
Splitting documents every N tokens is easy and wrong. Chunk along the document's own structure — sections, clauses, tables — and enrich each chunk with metadata: source, date, audience, jurisdiction.
Metadata powers filtered retrieval, and filtered retrieval is how you stop last year's policy from answering this year's question.
Hybrid retrieval beats embeddings alone
Embeddings capture meaning but miss exact identifiers — product codes, clause numbers, names. Keyword search nails identifiers but misses paraphrase. Production systems need both, plus a re-ranker to order the combined results.
Query rewriting helps too: expanding a terse user question into the vocabulary your corpus actually uses often lifts retrieval more than a better embedding model.
Design for honest failure
The most trust-building sentence a knowledge assistant can say is 'I can't find that in the documentation.' Confidence thresholds, citation requirements and explicit no-answer behaviour cost a little coverage and buy enormous trust.
Every answer should carry its sources. When users can click through and verify, adoption follows.