← back to the archiveCover illustration for “Reranking: a second, sharper pass over search results”
TERM8mo ago·Published · Updated ·by Andy Padia

Reranking: a second, sharper pass over search results

In short: Reranking reorders a shortlist of search results with a slower, more accurate model before the LLM reads them, trading latency and cost for precision.

Reranking is a second retrieval step that takes the top results from a fast first search, for example the best 100 hits, and reorders them with a slower, more accurate model that reads the query and each candidate together. Only the best few then go into the language model's context. Nogueira and Cho's paper from January 2019 applied BERT to this passage re-ranking step.

Reranking in plain words

A recruiter shortlists fifty CVs by keywords in minutes. A hiring manager then reads those fifty properly and picks the best five.

How it works

First-stage retrieval (BM25, vector or hybrid) is fast but scores the query and documents separately. A reranker, usually a cross-encoder model, scores each query-document pair jointly. The Sentence Transformers documentation credits cross-encoders with higher performance because they apply attention across the query and the document together. It also notes that scoring millions of pairs this way would be slow, which is why retrieval comes first.

The gain can be large. Nogueira and Cho reported that their BERT re-ranker beat the previous best MS MARCO passage-retrieval entry by 27% relative in MRR@10. Hosted services package the same step: Cohere's Rerank takes a query and a list of documents and returns them ordered by semantic relevance, each with a relevance score.

Example

As an illustration, the query "can contractors expense home internet?" retrieves 40 chunks about expenses. The reranker moves the one paragraph about contractor reimbursements from position 17 to position 1, and the answer becomes correct.

Where you'll meet it

  • Hosted rerank APIs such as Cohere Rerank
  • Open-source cross-encoder models
  • RAG frameworks' reranking stages
  • Search engines' learning-to-rank layers

What it changes for you

For developers, a reranker is worth testing before you swap the embedding model or the LLM, because it changes one stage and is easy to measure. For architects, it adds latency and per-query cost, so tune the shortlist size. For teams, it lets you send fewer chunks to the LLM, which cuts token spend and can reduce context rot.

Reranking vs retrieval

Retrieval finds candidates quickly across the whole corpus. Reranking judges a small set carefully. You need both.

Common mistake

My view: reranking a shortlist that does not contain the right answer. A reranker can only reorder what it is given. If recall is poor, fix the first stage with hybrid search or better chunking first.

Hybrid search · Retrieval-augmented generation · Chunking · Context rot · Semantic search

From the archive: Why your RAG pipeline returns irrelevant results · Before tuning retrieval, prove the missing item was indexed

A reranker can only reorder what the first stage found, so fix recall before you tune ranking.

#field-guide#rag#retrieval#search

Sources

  1. arxiv.org/abs/1901.04085
  2. sbert.net/examples/sentence_transformer/applications/retrieve_rerank/README.html
  3. docs.cohere.com/docs/rerank-overview
← older drop
Inference server: software that serves models at scale
newer drop →
Guardrails: checks that keep AI output in bounds

related drops

explore all 557 drops →
← back to the archiveday 175