---
title: "Reranking: why top-K retrieval isn't good enough"
description: "A fast retriever pulls top-N candidates, then a cross-encoder style reranker reads query and document together and reorders them. Traps fall, answers rise."
published: 2026-07-18
updated: 2026-07-18
canonical: https://protrailblazer.com/posts/retrieval-reranking/
tags: [machine-learning, demo, reranking, retrieval, rag, cross-encoder, semantic-search]
---

# Reranking: why top-K retrieval isn't good enough

**Answer:** Reranking is a second scoring pass that reads query and document together, catching relevance signals that a fast first-stage retriever misses by scoring them separately. A bi-encoder or keyword index retrieves a broad candidate set at speed, then a cross-encoder style reranker reads each (query, document) pair jointly and reorders the list. Documents that looked relevant on surface overlap fall; documents with precise intent or phrase alignment rise. The demo lets you watch that reorder happen candidate by candidate, with a feature breakdown showing exactly why each document moved.

The interactive below runs a two-stage retrieval pipeline over a 36-document corpus. Pick a query preset, drag the Retrieve N and Rerank depth sliders, toggle the reranker on and off, and tap any candidate row to read the full feature breakdown that drove its score.

*Interactive demo: [open the reranking demo](https://protrailblazer.com/demos/reranking/)*

## What you're seeing

The scene runs left to right in two stages. On the left, the stage-1 panel streams the full corpus through a fast retriever that scores each document in isolation using token overlap and a hand-authored semantic feature. The top-N documents by that score become the candidate list, shown as a vertical ranked column with each document's title, snippet, and initial rank.

When the animation reaches the reranking pass, the selected candidates advance to stage 2. A cross-encoder style scorer reads query and document together and computes pairwise features: intent alignment, phrase match, field weight, and negation detection. Rank-change arrows appear between the two columns as documents reshuffle. A document that matched on keywords but triggered a negation feature drops; a document with low surface overlap but strong intent alignment rises. The final top-K winners land in the answer context tray on the right.

The HUD across the top tracks how many candidates were retrieved, how many the reranker read, the simulated pipeline latency and estimated token count (both labeled SIMULATED), and REL@K, the share of ideal hand-labeled relevance that made it into the final top K. The candidate inspector (tap any row) shows the exact feature breakdown and a note when a special feature such as negation fired.

A few things to try:

- **Switch to SUPPORT query and watch the negation trap.** In balanced mode, "Delete your account and erase all data" ranks highly after stage 1 on keyword overlap with account-management docs. After reranking it falls sharply. Tap its row at each stage to compare the negation feature before and after the joint read.
- **Toggle the reranker off with Reranker On/Off.** The final tray fills from the stage-1 list unchanged. Compare REL@K with the reranker on versus off to see how much precision the second pass adds for the same N.
- **Switch between SPEED, BALANCED, and QUALITY mode presets.** SPEED turns the reranker off entirely (RERANKED 0, TOP-5 Δ 0). QUALITY reranks all 30 retrieved candidates at higher simulated latency and token cost. Watch the LATENCY and TOKENS READ stats update to see the cost of a deeper rerank.
- **Drag Rerank depth down to match Final K.** Setting depth to 5 with K=5 means the reranker reads only the documents it must return. REL@K often stays near the full-depth value because the most relevant documents tend to cluster near the top of the stage-1 list.
- **Try the COMPARE and POLICY presets for different failure modes.** COMPARE has queries where the key signal is a field match (pricing tier, feature table), not keyword density. POLICY queries have low surface overlap with the most relevant answer, so intent alignment does most of the heavy lifting in stage 2.

## How two-stage retrieval actually works

The problem reranking solves is a speed-quality tradeoff baked into the first stage. A dense retriever encodes the query into a vector, encodes every document separately, and finds nearest neighbors by dot product or cosine distance. This is fast because the document embeddings are precomputed and the query encoding is a single forward pass. But scoring is done in isolation: the model never sees query and document at the same time, so fine-grained signals like intent, negation, or phrase alignment cannot influence the score.

A cross-encoder reranker fixes this by concatenating the query and document into a single input and running a joint scoring pass. It can see that 'delete account' and 'manage account' share most of their tokens but have opposite intent. The price is linear cost: scoring N candidates requires N separate forward passes through the reranker. That is acceptable for a shortlist of 10 to 30 documents; it would be prohibitive for the full corpus.

This is why two stages exist. Stage 1 is a recall gate: fast, approximate, designed to ensure the correct documents are somewhere in the top-N candidates. Stage 2 is a precision pass: slower, exact, designed to put the best documents at the top of the final K returned to the user or the LLM.

The demo's scoring model is a deterministic teaching version, not a real neural network. It computes hand-authored features for token overlap, a semantic category match, an intent alignment score, a phrase match bonus, a field weight (metadata like pricing or API docs), and a negation penalty when the query contains a negation word paired with a matching concept. These features mimic the kinds of signals a real cross-encoder would learn from training data, labeled clearly as SIMULATED in the UI.

## Why rerank depth is a design choice, not a fixed setting

Reranking depth is the number of stage-1 candidates the reranker reads. Setting depth equal to final K is the minimum: the reranker scores exactly the documents it will return, at the lowest possible token cost. Setting depth higher increases the chance the reranker will promote a document that stage 1 ranked below K but that has strong pairwise signals.

In the demo you can drag depth from K up to N and watch TOKENS READ rise proportionally. REL@K typically plateaus well before full depth because the very best documents tend to land near the top of the stage-1 list anyway, and the marginal gain from reading further diminishes. The practical sweet spot in production systems is usually 2x to 4x K, which recovers most of the quality improvement at a fraction of the full-depth cost.

The same tradeoff applies to latency. A cross-encoder running on GPU can score 10 to 30 documents quickly enough that the added latency is small compared to the LLM generation step that follows. That math changes for very long documents, where the context window cost of concatenating query and document grows with document length. This is one reason late-interaction models like ColBERT, which precompute token-level embeddings rather than full text, are appealing for deep reranking over long documents.

To see how the documents that survive reranking get consumed by an LLM, the [retrieval-augmented generation walkthrough](https://protrailblazer.com/posts/retrieval-augmented-generation) traces the full RAG pipeline. And for the vector space that the stage-1 retriever navigates, see [embeddings in 3D](https://protrailblazer.com/posts/embeddings-in-3d).

## Frequently asked questions

### What is the difference between a bi-encoder and a cross-encoder?

A bi-encoder encodes query and document independently and compares the resulting vectors, usually with dot product or cosine distance. It is fast because document vectors can be precomputed. A cross-encoder concatenates query and document into one input and scores the pair jointly, giving it access to fine-grained interaction signals at the cost of requiring one forward pass per candidate. Most production systems use a bi-encoder for stage 1 and a cross-encoder for reranking.

### What is REL@K and why does it matter more than the ranked list order?

REL@K measures how much of the ideal hand-labeled relevant set appears in the final top K results, expressed as a percentage. A pipeline can produce a nicely ordered top-5 list while missing one of the two genuinely relevant documents entirely, which would show as REL@5 of 50 percent rather than 100 percent. Rank order within the top K matters for presentation but REL@K tells you whether the retrieval found the right things at all.

### Does reranking help keyword search as well as vector search?

Yes. Two-stage pipelines pair a reranker with BM25 or TF-IDF just as often as with a dense vector retriever. The stage-1 method changes; the stage-2 reranker does not care how the candidates were retrieved. Hybrid systems that combine keyword and dense retrieval in stage 1, then rerank in stage 2, are common in production because they capture recall advantages from both approaches.

### What happens when I toggle the reranker off in the demo?

With the reranker off, the final top-K documents are taken directly from the stage-1 ranking by score. The demo is in SPEED mode and the RERANKED stat shows 0. You can compare REL@K with the reranker on versus off at the same N and K to see how much precision the second pass adds. For simple queries where the stage-1 retriever already surfaces the right documents in the right order, the gap is small. For queries with negation or intent ambiguity, it can be large.

### Are the latency and token numbers in the demo realistic?

No, they are simulated to illustrate relative cost, not absolute performance. The HUD labels them SIMULATED. Real reranker latency depends on the model architecture, hardware, batching strategy, and document length. A dedicated cross-encoder on GPU can rerank 20 short documents in under 50ms; a large LLM used as a reranker on CPU could take seconds. The demo uses the numbers to show that depth and N are levers on cost, not to make claims about any specific system.

## Key takeaways

- Stage-1 retrievers score query and document independently, which makes them fast enough to scan thousands of documents but blind to intent signals that only appear when both texts are read together.
- A document can rank highly on keyword overlap while being semantically wrong: a negation like 'delete your account' shares tokens with 'manage your account' and climbs stage 1, but the cross-encoder style reranker catches the negation feature and drops it.
- Reranking depth is a latency knob, not a quality absolute: reranking only the top 10 of 20 candidates costs half the token budget of reranking all 20, and the quality gap narrows quickly because the worst candidates rarely hold genuinely relevant documents.
- REL@K measures what actually matters: how much of the ideal relevance set survived into the final top K. Simulated latency and token counts show the cost side of the same tradeoff.
- Turning the reranker off and switching to SPEED mode shows that the stage-1 list is not random noise: it already contains most relevant documents, just in the wrong order. Reranking is a precision move, not a recall move.