Why your RAG pipeline returns irrelevant results
In short: Debug a RAG pipeline by tracing one wrong answer to the first failed evidence handoff, from ingestion through chunking, ranking and context to the citation.
Anthropic reported on 19 September 2024 that adding context to each chunk before embedding, combining the result with BM25 and reranking the candidates cut its top-20 retrieval failure rate from 5.7% to 1.9%. That is a vendor result, not a universal gain — and still the most useful number in the RAG literature, because it names where answers die: before the model ever sees the passage.
That is the method for a RAG pipeline that returns irrelevant results. Do not tune the generator first. Take one wrong answer and follow the evidence through every handoff — ingestion, chunking, access filtering, retrieval, reranking, context assembly, generation, citation — until you find the first stage where the right passage was lost. Repair that stage, re-run, and only then look at the next one. A better prompt cannot recover a passage that was split in half at indexing time, and a reranker cannot promote a document the access filter removed.
The trace that makes the failure visible
Most pipelines cannot answer "which chunk did the model see, and from which version of the document?" — and that is the bug behind the bug. The trace below is the minimum record per request; without it, debugging is guesswork.
Each box writes one line to the trace: the source identifier and version at ingest, the chunk IDs and their boundaries, the eligible set after filters, candidate scores, final ranks, the exact text passed as context, and for every claim in the answer the passage it rests on. The last line is the one teams skip, and it is the one that tells you whether an answer was grounded or merely accompanied by documents.
Symptom, test, repair
| Symptom | Test (one change at a time) | Likely stage | Repair |
|---|---|---|---|
| Right document, wrong passage | find the answer span in the raw source; check which chunk holds it | chunking | re-chunk on structure; keep tables and headed sections whole; add chunk context |
| Right facts, wrong version | compare cited source version with the current one | ingestion | version-aware index; evict or down-rank superseded sources |
| Document never appears | query the index directly for the known chunk ID | access filter or recall | fix eligibility first — recall is a corpus-scope problem — then hybrid retrieval |
| Right chunk retrieved, ranked too low | inspect candidate ranks before and after reranking | ranking | add or fix the reranker; check score normalisation across hybrid sources |
| Chunk in context, ignored | diff passed context against the answer's citations | assembly or generation | shorten context, order by rank, require citations per claim |
| Confident answer, no support | run the claim-to-passage check | citation | answer must cite or decline; treat "no support" as a failure class |
| Answer when the corpus has none | ask a question with no answer in the fixture | generation | an explicit "not in the sources" path, graded as a pass |
Measure retrieval and answer quality separately: a context-recall number and an answer-rating number, as OpenAI's document-Q&A evaluation example separates them. Its example thresholds are illustrations, not launch standards.
The fixture you can inspect
A RAG repair needs a corpus small enough to read. Build a public-document fixture on purpose: a dozen documents with exact identifiers in the text, one table split across a chunk boundary, one document present in two versions, one question whose answer is absent, and one document the test user is not authorised to see. Those five properties cover the stages in the table. Then change one thing at a time — chunk boundaries, the hybrid weighting, the reranker, the context budget — and re-run the same questions, recording context recall and answer rating after each change. Anthropic's 5.7% to 1.9% was produced that way, with each technique added in sequence; the method transfers even where the numbers will not.
This is the stage-by-stage counterpart to two decisions elsewhere on this site: whether retrieval is the right intervention at all, and what retrieval changes versus what fine-tuning changes. Here the decision has been made; the question is where the pipeline is dropping the evidence.
The lab I would run
No retrieval experiment was executed for this piece; the fixture above is a proposed lab, not a client story, and no optimal chunk size or expected gain is claimed. The run: take one wrong support answer from your logs. Pull its trace — or, if there is no trace, build the eight-line record first and reproduce the request. Locate the answer span in the raw source, find its chunk, check the chunk was eligible, retrieved, ranked and passed, and check the citation line. The first "no" is the stage to fix. Expect, in most pipelines I have reviewed the shape of, that the first "no" is at chunking or at the access filter, and that the generator was innocent.
Where this holds and where it breaks
It holds for document-grounded features with a finite corpus — policies, manuals, tickets, contracts — where a wrong answer can be traced to a passage. It is weaker for open-web retrieval, where the corpus changes under you and version is a moving target. It breaks without a trace; a pipeline that cannot say which chunks it passed cannot be debugged, only re-prompted, and the first repair is the trace itself — the same pre-production evaluation discipline that applies to any LLM feature.
FAQ
What chunk size should I use?
The one your fixture proves. There is no universal size; split on document structure, keep tables and headed sections intact, and test the boundary cases you built on purpose.
Hybrid search or embeddings alone?
Test both on your fixture with exact identifiers in the questions. Lexical retrieval tends to win on identifiers and codes; embeddings on paraphrase. Anthropic's reported gain came from combining them with reranking.
Does a citation prove the answer is grounded?
Only if the citation is checked claim by claim against the passage. A citation attached to a paragraph proves the document was in context, not that the claim came from it.
What's in it for you
- An eight-line trace record that makes every evidence handoff in a RAG pipeline inspectable.
- A seven-row symptom-to-test-to-repair table that locates the failed stage before anyone touches the prompt.
- A five-property fixture design you can build from public documents and change one variable at a time.
Trace one wrong answer to the first stage that lost the passage and fix that stage; the generator is usually the last place a RAG pipeline fails, and the first place teams look.
