← back to the archiveCover illustration for “Chunking: splitting documents for retrieval and RAG”
TERM6mo ago·Published · Updated ·by Andy Padia

Chunking: splitting documents for retrieval and RAG

In short: Chunking splits documents into passages before they are embedded for search and RAG, and the split you choose changes what gets retrieved.

Chunking is splitting documents into smaller passages before indexing them for search or RAG. Each chunk is embedded and retrieved on its own, so chunk boundaries decide what the model can see together. Too small and chunks lose meaning; too large and they dilute relevance and waste context. Pinecone's guide, updated in June 2025, says there is no one-size-fits-all answer.

Chunking in plain words

Think of cutting a book into index cards. Cut mid-sentence and each card is useless; make each card a whole chapter and you cannot find the one paragraph you need.

How it works

Common strategies include:

  1. Fixed size with overlap. Pinecone suggests testing sizes such as 128, 256, 512 and 1,024 tokens against your own queries.
  2. Structure-aware: split on headings, sections, paragraphs or code functions.
  3. Semantic: split where topic shifts, detected with embeddings.
  4. Contextual: prepend a short description of where the chunk sits in the document, so a chunk saying "the limit is 30 days" keeps the fact that it is about refunds.

Anthropic reported in September 2024 that adding 50 to 100 tokens of chunk-specific context cut failed retrievals by 35%, and by 49% when combined with contextual BM25.

Example

As an illustration, an HR policy split every 500 characters separates "Employees in India are eligible for" from "26 weeks of maternity leave". Splitting on headings keeps the whole "Maternity leave in India" section together, and retrieval returns a complete answer.

Where you'll meet it

  • Document loaders and text splitters in RAG frameworks
  • Ingestion pipelines for PDFs, wikis and tickets
  • Code search, where chunks follow functions or classes
  • Retrieval evaluation, where chunking is a main tuning knob

What it changes for you

For developers, chunking is where domain knowledge pays off: legal clauses, API docs and chat logs each need different splits. For data teams, re-chunking means re-embedding the corpus, so test before scaling. My view: better chunking often improves answers more than a model switch does.

Chunking vs tokenisation

Tokenisation splits text into tokens for the model to read. Chunking splits documents into passages for retrieval. They sit at different layers and serve different goals.

Common mistake

My view: picking a chunk size from a tutorial and never testing it. Build twenty real questions with known answers, try three chunking strategies, and keep the one that retrieves the right passage most often.

Retrieval-augmented generation · Embeddings · Reranking · Context window · Golden dataset

From the archive: Why your RAG pipeline returns irrelevant results

Test chunking strategies on real questions with known answers before you scale the index.

#field-guide#rag#retrieval#data-preparation

Sources

  1. pinecone.io/learn/chunking-strategies
  2. anthropic.com/research/contextual-retrieval
← older drop
BM25: the keyword ranking formula behind search
newer drop →
Agentic coding: AI agents doing the coding, you owning it

related drops

explore all 557 drops →
← back to the archiveday 175