← back to the archiveCover illustration for “RAG: retrieval-augmented generation explained”
TERMday 162·13d ago·Published · Updated ·by Andy Padia

RAG: retrieval-augmented generation explained

In short: RAG retrieves relevant documents at query time and gives them to an LLM so answers draw on your data: how it works, an example and why it fails.

Retrieval-augmented generation (RAG) is a pattern where a system first searches a collection of documents for passages relevant to a question, then gives those passages to a language model to answer from. It lets models use private or recent information they were never trained on, and lets answers cite sources. The term comes from a 2020 Facebook AI research paper.

RAG in plain words

An open-book exam. The model does not need to have memorised your company handbook; it looks up the right pages, then writes the answer.

How it works

  1. Index: split documents into chunks, create embeddings, store them in a vector index (often alongside a keyword index).
  2. Retrieve: turn the question into a query, find the most relevant chunks.
  3. Rerank (optional): reorder candidates with a more precise model.
  4. Generate: put the top chunks in the prompt and ask the model to answer only from them, with citations.

The original design, as Meta's September 2020 write-up describes it, had the same shape: retrieve documents for the input, then pass both to a generator.

Example

As an illustration, "What is our parental leave policy in India?" retrieves the two HR policy sections tagged India, and the model answers in three sentences citing section numbers. Without retrieval, it would guess from general knowledge.

Where you'll meet it

  • Internal knowledge assistants and support bots
  • Documentation search with AI answers
  • Enterprise search products and AI features in SaaS tools
  • AI search engines that cite web pages

What it changes for you

For developers, answer quality depends far more on retrieval (chunking, search, reranking) than on the model. For data teams, document hygiene, permissions and freshness become AI problems. For leaders, RAG is usually the first enterprise AI system worth building, and the first to fail without anyone noticing when content goes stale.

RAG vs fine-tuning

RAG gives the model the right facts at question time. Fine-tuning changes how the model behaves. Use RAG for knowledge, fine-tuning for style or skills.

Common mistake

My view: blaming the model when RAG answers are wrong. Check whether the right passage was retrieved at all. A wrong RAG answer usually starts as a retrieval miss, so debug the search before the prompt.

Embeddings · Vector database · Hybrid search · Reranking · Chunking · Agentic RAG

From the archive: Why your RAG pipeline returns irrelevant results · Fine-tune to change behaviour, use RAG to change knowledge

When a RAG answer is wrong, check what was retrieved before you touch the prompt.

#field-guide#rag#retrieval#llm#search

Sources

  1. arxiv.org/abs/2005.11401
  2. ai.meta.com/blog/retrieval-augmented-generation-streamlining-the-creation-of-intelligent-natural-language-processing-models
← older drop
Context window: how much an LLM can read at once
newer drop →
Model Context Protocol (MCP): a USB-C port for AI tools

related drops

explore all 557 drops →
← back to the archiveday 175