← back to the archiveCover illustration for “Agentic RAG: an agent that decides what to search for”
TERM6mo ago·Published · Updated ·by Andy Padia

Agentic RAG: an agent that decides what to search for

In short: Agentic RAG lets an AI agent plan searches, choose sources, judge results and search again before answering, at the cost of more model calls and latency.

Agentic RAG is retrieval-augmented generation where an AI agent controls retrieval instead of a fixed pipeline. The agent decides whether to search, which source or tool to use, how to phrase queries, whether the results are good enough and when to search again before it answers. An early research form is Self-RAG, submitted in October 2023, which retrieves passages on demand and critiques them.

Agentic RAG in plain words

Standard RAG is one trip to the library with one request slip. Agentic RAG is a researcher who goes back for more books when the first ones do not answer the question.

How it works

Search becomes a tool in an agent loop. The agent breaks the question into sub-questions, searches each (possibly across a vector index, SQL, the web or APIs), checks whether results actually answer it, rewrites queries if not, then synthesises with citations.

Anthropic's December 2024 guide to building agents describes the base unit as an LLM augmented with retrieval and tools that generates its own search queries. The same guide recommends stopping conditions, such as a maximum number of iterations, which is what keeps the agent from searching forever.

Example

As an illustration, "How did our Q3 cloud spend compare with budget, and which team drove the overrun?" The agent queries the finance database for totals, searches the budget document, then queries cost-allocation data by team, and only then answers. One-shot RAG would retrieve a few loosely related chunks.

Where you'll meet it

  • Deep-research features in assistants
  • Enterprise assistants spanning documents, databases and APIs
  • Coding agents searching codebases iteratively
  • Research on self-reflective retrieval such as Self-RAG

What it changes for you

For developers, retrieval tools need clear descriptions and good error messages, because the agent chooses among them. For product teams, multi-part and ambiguous questions are where the extra searches earn their keep, and every one of them adds latency, so route simple questions to plain RAG. For cost owners, track searches per answer as a metric.

Agentic RAG vs standard RAG

Standard RAG retrieves once with the user's question and answers. Agentic RAG plans, retrieves repeatedly and checks its evidence.

Common mistake

My view: making every question agentic. Most questions are simple lookups where an extra planning step only adds delay and cost. Use a router: simple questions go to standard RAG, complex ones to the agent.

Retrieval-augmented generation · AI agent · Tool calling · Model routing · GraphRAG

From the archive: Why your RAG pipeline returns irrelevant results

Route simple lookups to standard RAG and keep the agent for questions that need several searches.

#field-guide#rag#agents#retrieval

Sources

  1. arxiv.org/abs/2310.11511
  2. anthropic.com/research/building-effective-agents
← older drop
Build vs buy (AI): when to make your own AI and when not
newer drop →
MCP server: how a tool exposes itself to AI apps

related drops

explore all 557 drops →
← back to the archiveday 175