
Reranking: a second, sharper pass over search results
In short: Reranking reorders a shortlist of search results with a slower, more accurate model before the LLM reads them, trading latency and cost for precision.
Reranking is a second retrieval step that takes the top results from a fast first search, for example the best 100 hits, and reorders them with a slower, more accurate model that reads the query and each candidate together. Only the best few then go into the language model's context. Nogueira and Cho's paper from January 2019 applied BERT to this passage re-ranking step.
Reranking in plain words
A recruiter shortlists fifty CVs by keywords in minutes. A hiring manager then reads those fifty properly and picks the best five.
How it works
First-stage retrieval (BM25, vector or hybrid) is fast but scores the query and documents separately. A reranker, usually a cross-encoder model, scores each query-document pair jointly. The Sentence Transformers documentation credits cross-encoders with higher performance because they apply attention across the query and the document together. It also notes that scoring millions of pairs this way would be slow, which is why retrieval comes first.
The gain can be large. Nogueira and Cho reported that their BERT re-ranker beat the previous best MS MARCO passage-retrieval entry by 27% relative in MRR@10. Hosted services package the same step: Cohere's Rerank takes a query and a list of documents and returns them ordered by semantic relevance, each with a relevance score.
Example
As an illustration, the query "can contractors expense home internet?" retrieves 40 chunks about expenses. The reranker moves the one paragraph about contractor reimbursements from position 17 to position 1, and the answer becomes correct.
Where you'll meet it
- Hosted rerank APIs such as Cohere Rerank
- Open-source cross-encoder models
- RAG frameworks' reranking stages
- Search engines' learning-to-rank layers
What it changes for you
For developers, a reranker is worth testing before you swap the embedding model or the LLM, because it changes one stage and is easy to measure. For architects, it adds latency and per-query cost, so tune the shortlist size. For teams, it lets you send fewer chunks to the LLM, which cuts token spend and can reduce context rot.
Reranking vs retrieval
Retrieval finds candidates quickly across the whole corpus. Reranking judges a small set carefully. You need both.
Common mistake
My view: reranking a shortlist that does not contain the right answer. A reranker can only reorder what it is given. If recall is poor, fix the first stage with hybrid search or better chunking first.
Related terms
Hybrid search · Retrieval-augmented generation · Chunking · Context rot · Semantic search
From the archive: Why your RAG pipeline returns irrelevant results · Before tuning retrieval, prove the missing item was indexed
A reranker can only reorder what the first stage found, so fix recall before you tune ranking.


