Skip to content

Advanced RAG — Post-retrieval Techniques

Retrieval doesn't end when you have your top-k chunks. Naive retrieval's ranking is only ever approximate, and even a genuinely relevant chunk is usually padded with sentences that don't answer the question. The techniques on this page all operate after the initial search — re-scoring what came back, trimming it down, or both — before any of it reaches the LLM.

Hands-on

Every technique below is runnable, and every number is measured, not asserted: Post-retrieval → in the code repo. Cross-encoder reranking needs no LLM key (fully local); contextual compression and listwise reranking both need one.

TL;DR — quick revision
  • Cross-encoder reranking fixes a real miss: naive bi-encoder retrieval finds the right chunk somewhere in a wider candidate pool but ranks it outside the top-3; reranking with a cross-encoder correctly promotes it — because it reads query and chunk together, instead of comparing two independently-computed vectors
  • Contextual compression cuts ~94% of the retrieved text on a typical multi-chunk retrieval, keeping only the sentences that actually answer the question — same underlying fact, far less padding sent to the final LLM call
  • LLM-based listwise reranking (RankGPT-style) fixes the same miss cross-encoder reranking does — but at roughly 230x the latency (one full LLM round-trip vs. one local forward pass), since it sends the whole candidate list to the LLM in a single call instead of scoring each candidate independently
  • A real prompting lesson worth knowing: telling the model to "return ONLY the list" doesn't work — it explains its reasoning anyway. Fighting that after the fact (parsing around the prose) is fragile; asking it to wrap just the final answer in an unambiguous <ranking>...</ranking> tag and letting it reason freely everywhere else is the fix that actually holds up
  • The overarching lesson: reranking's value depends on naive retrieval actually having a ranking problem — recall the chunk, rank it wrong. On a small, well-organized corpus, naive bi-encoder retrieval is often already good enough that there's nothing to fix; reranking earns its keep on messier, larger corpora with more distractor chunks

Cross-Encoder Reranking

The problem it fixes: bi-encoder retrieval (what naive RAG does) embeds the query and every chunk independently, then compares vectors with cosine similarity. That independence is exactly what makes it fast and scalable — chunk embeddings are pre-computed once, offline, and search is just nearest-neighbor lookup. But it also means the model never looks at the query and a chunk together. It can retrieve the right chunk into a wide candidate pool and still rank it below several chunks that are only superficially similar.

A cross-encoder fixes this by feeding query and chunk into the model together — [CLS] query [SEP] chunk [SEP] — so bidirectional attention lets every query token attend to every chunk token and vice versa in one forward pass. It outputs a single relevance score per pair. This is far more accurate, but nothing can be pre-computed: every query-chunk pair needs a fresh forward pass, which is too slow to run over an entire corpus. The standard production pattern is therefore two-stage: bi-encoder retrieval casts a wide net (e.g. top-50 to top-150, optimizing for recall), then a cross-encoder rescoring only that shortlist (optimizing for precision). This is exactly Anthropic's Contextual Retrieval recommendation: hybrid retrieval → top-150 → cross-encoder rerank → top-20 → LLM.

from sentence_transformers import CrossEncoder

CROSS_ENCODER_MODEL_NAME = "cross-encoder/ms-marco-MiniLM-L-6-v2"


def cross_encoder_rerank(question: str, candidates: list[str], top_n: int = 3) -> list[str]:
    """Bi-encoder retrieval casts a wide net (candidates); the
    cross-encoder reads query+chunk TOGETHER in one forward pass per
    pair (bidirectional attention, no pre-computation possible) and
    outputs a real relevance score per candidate — then keeps only the
    top_n. Too expensive to run over a whole corpus, fine over an
    already-narrowed candidate set."""
    reranker = CrossEncoder(CROSS_ENCODER_MODEL_NAME)
    pairs = [(question, c) for c in candidates]
    scores = reranker.predict(pairs)
    ranked = [c for _, c in sorted(zip(scores, candidates), key=lambda x: -x[0])]
    return ranked[:top_n]

Input → output, from an actual run:

Input question:
  What is the complexity per layer for a recurrent layer compared to
  self-attention?

Naive bi-encoder retrieval (15-candidate pool), target chunk ranked #4
— outside the top-3:
  [rank 0] "...As noted in Table 1, a self-attention layer connects all
            positions with a constant number of sequentially executed
            operations, whereas a recurrent layer requires O(n)..."
  [rank 1] "...recurrent layers, by a factor of k. Separable convolutions
            [6], however, decrease the complexity considerably..."
  [rank 2] "...layer in a typical sequence transduction encoder or
            decoder. Motivating our use of self-attention we consider
            three desiderata..."
  [rank 3] "Table 1: Maximum path lengths, per-layer complexity and
            minimum number of sequential operations..." <- the actual
            table containing O(n·d2) for recurrent layers, MISSED by
            naive top-3

Cross-encoder rescoring the same 15 candidates (relevance scores, not
ranks — notice the big drop after rank 2):
  rank 0: score=7.07
  rank 1: score=6.30
  rank 2: score=4.55  <- target chunk, now correctly in the top-3
  rank 3: score=4.23
  rank 4: score=3.21
  ...
  rank 14: score=-6.38

Measured: fixes a real miss

Naive top-3 never surfaces the chunk containing "O(n·d2)" (the recurrent-layer complexity from Table 1) — it's present in the wider 15-candidate pool, just ranked 4th. Reranking that same pool with a cross-encoder correctly promotes it into the top-3. The cross-encoder's own scores show why: it gives the target chunk a clearly positive score (4.55) while everything past rank 5 or so scores negative — it isn't subtly reordering close calls, it's confidently separating genuinely relevant chunks from ones that only looked relevant to the bi-encoder.

A caveat worth taking seriously: the ARAGOG benchmark found reranking doesn't show a clear advantage over naive retrieval on every dataset. That tracks with what's demonstrable here too — most already-clear questions against this small, well-organized paper already have naive bi-encoder retrieval finding the right chunk in the top-3, leaving nothing for a reranker to fix. Reranking's value shows up specifically when naive retrieval's recall is fine but its ranking isn't — a messier, larger corpus with more genuinely similar-looking distractor chunks will show this more often than a single 15-page paper does.

Use it for: production pipelines where you already do wide bi-encoder retrieval and want to sharpen the final top-n before generation — cheap once you have a candidate pool, since scoring 15 candidates takes well under a tenth of a second with the model already loaded. Skip it for: corpora where naive retrieval's top-k is already reliable — measure first, same principle as every other technique on this site.

How a cross-encoder actually produces that single score

It's not a repurposed next-token-prediction model. Most everyday LLMs (GPT, LLaMA) are decoder-only, trained via next-token prediction with causal attention — each token only attends to previous tokens, which is what makes next-token prediction work at all. A cross-encoder is usually a different architecture family entirely: encoder-only, BERT-style, with bidirectional attention — every token attends to every other token, in both directions, with no next-token constraint. That's what lets it deeply model the full relationship between every query token and every chunk token in one pass.

The input is formatted as [CLS] query tokens [SEP] chunk tokens [SEP]. The [CLS] token's final hidden state is treated as a pooled summary of the whole pair (standard BERT practice), and a small classification head (1-2 linear layers) turns that vector into a single relevance score — often passed through a sigmoid for a 0-1 "how relevant" probability.

Training is supervised, not next-token prediction: labeled (query, chunk, relevance) triples, usually via pointwise loss (binary cross-entropy — relevant or not) or pairwise loss (given a relevant and an irrelevant chunk for the same query, train score(relevant) > score(irrelevant) by a margin — common, since ranking is fundamentally about relative order). Training data commonly comes from MS MARCO (real search-engine query-relevance pairs, hundreds of thousands of examples), fine-tuned on top of a pretrained BERT-style checkpoint.

You don't train your own — pretrained rerankers already exist and are what production systems actually use: cross-encoder/ms-marco-MiniLM-L-6-v2 (sentence-transformers, open-source, the one used in this page's demo), Cohere Rerank (hosted API, the one ARAGOG benchmarked), BGE-reranker (BAAI, open-source, multiple sizes), and Jina Reranker (hosted/open). Newer rerankers are increasingly built on decoder-only LLMs directly (prompted for a relevance score), but classic BERT-style cross-encoders remain the production default since they're far cheaper per-pair than a full generative call — a cost/latency trade-off that comes up again below, when listwise LLM reranking is measured directly against this same cross-encoder.

Contextual Compression

The problem it fixes: even a genuinely relevant retrieved chunk is usually mostly padding relative to the question — a 400-500 token chunk might have one sentence that actually answers it. That padding costs tokens, and it can actively hurt quality via the "lost in the middle" effect: LLMs pay less attention to information buried in the middle of a long context than to what's near the beginning or end, so stuffing several long, mostly-irrelevant chunks into the context window can degrade the final answer, not just its cost.

Contextual compression runs each retrieved chunk through a cheap LLM call that extracts (or rewrites) only the sentences relevant to the question, discarding the rest — after retrieval (and optionally after reranking), before the final generation call.

import llm

COMPRESS_PROMPT = """From the passage below, extract ONLY the sentences \
that are relevant to answering the question. Return them verbatim, \
nothing added. If nothing in the passage is relevant, return "NONE".

Question: {query}

Passage:
{chunk}"""


def contextual_compress(question: str, chunks: list[str], provider: str | None = None) -> list[str]:
    """Runs one compression call per retrieved chunk (independent of
    each other — parallelizable in a real deployment) using a cheap
    LLM call, trimming each chunk down to just the sentences relevant
    to the question. Chunks that compress to "NONE" are dropped."""
    compressed = []
    for chunk in chunks:
        result = llm.generate(COMPRESS_PROMPT.format(query=question, chunk=chunk), provider=provider).strip()
        if result and result.upper() != "NONE":
            compressed.append(result)
    return compressed

Input → output, from an actual run:

Input question: What optimizer did they use for training?

5 chunks retrieved (98-484 chars each, ~2000 chars total) — only 2 of
the 5 actually contain anything about the optimizer:

  chunk 0 (98 chars):  "warmup_steps = 4000. 5.4 Regularization We employ
                        three types of regularization during training..."
  chunk 1 (484 chars): "(3.5 days). 5.3 Optimizer We used the Adam
                        optimizer [20] with β1 = 0.9, β2 = 0.98 and
                        ϵ = 10−9. We varied the learning rate..."
  chunk 2 (453 chars): "target tokens. 5.2 Hardware and Schedule We
                        trained our models on one machine with 8 NVIDIA
                        P100 GPUs. For our base models..."
  chunk 3 (484 chars): "Table 2: The Transformer achieves better BLEU
                        scores than previous state-of-the-art models..."
  chunk 4 (461 chars): "were written at 10-minute intervals. For the big
                        models, we averaged the last 20 checkpoints..."

After compression — chunks 0, 3, 4 compressed to "NONE" and dropped
entirely; chunks 1 and 2 kept only their relevant sentence:

  "We used the Adam optimizer [20] with β1 = 0.9, β2 = 0.98 and ϵ = 10−9."
  "We trained our models on one machine with 8 NVIDIA P100 GPUs."

Measured: ~94% reduction, target fact survives

The 5 retrieved chunks total ~1980 characters; compression brings that down to ~128 characters across 2 surviving extracts — a 94% reduction — and the actual answer ("Adam") is still in there. Notice this isn't just trimming: 3 of the 5 chunks compress to nothing at all, because they were retrieved (correctly, by similarity) as being on-topic but don't actually answer this specific question. Reranking would have kept or dropped these chunks wholesale; compression operates at the sentence level, within chunks reranking would have kept.

When it's actually worth the extra call: for a single retrieved chunk, skip it — compress-then-generate is two calls where generate-directly is one, for no benefit. It earns its keep specifically on multi-chunk retrieval (the common case, top-n = 5 to 20), for three real reasons: the per-chunk compression calls run in parallel using a cheap model, so added latency is closer to "one parallel batch + one final call" than pure sequential addition; the expensive high-quality model then runs once on a much smaller context instead of paying premium prices for every token of every raw chunk; and a smaller, higher-signal context directly mitigates "lost in the middle."

Use it for: multi-chunk retrieval where individual chunks are long relative to what actually answers the question. Skip it for: single-chunk retrieval, or chunks that are already tightly scoped (small, focused chunking strategies leave less padding for compression to trim).

LLM-Based Listwise Reranking

The problem it fixes — or rather, a different way of fixing the same problem: a cross-encoder scores each candidate chunk independently against the query (pointwise). An LLM can instead see the entire candidate list at once and reason about candidates relative to each other in a single call — this is the RankGPT approach: number the candidates, ask the LLM to return them reordered by relevance. It's a genuinely more modern technique than classic BERT-style cross-encoders, and worth knowing as an alternative, not a strict upgrade.

import re

import llm

LISTWISE_PROMPT = """You are ranking passages by relevance to a question.

Question: {query}

Passages:
{numbered_passages}

Think through which passages are most relevant if you like. When you \
are done, output your final ranking as passage numbers from MOST to \
LEAST relevant, wrapped in <ranking></ranking> tags, with numbers \
separated by commas and every number included exactly once.

Example: <ranking>3,1,4,2</ranking>"""


def llm_listwise_rerank(question: str, candidates: list[str], top_n: int = 3, provider: str | None = None) -> list[str]:
    """RankGPT-style listwise reranking: instead of scoring each
    candidate independently (pointwise, like a cross-encoder), the LLM
    sees the WHOLE candidate list at once and outputs a full relevance
    ordering in one call — it can reason about candidates relative to
    each other, not just each one in isolation. Costs one LLM call for
    the whole list instead of one lightweight forward pass per
    candidate, and typically has much higher latency than a
    cross-encoder for a comparable-size candidate set.

    An earlier version of this prompt instructed the model to "return
    ONLY the list, nothing else" — the model routinely ignored that and
    explained its reasoning first, and a parser trusting the whole
    response to be just numbers picked up stray digits from that prose
    ("Table 1", "[4]") and silently corrupted the ranking. Fighting a
    model's tendency to explain itself is a losing battle; instead this
    prompt lets it reason freely and asks it to wrap ONLY the final
    answer in an unambiguous <ranking> tag — trivial to extract
    correctly regardless of how much reasoning precedes it."""
    numbered = "\n".join(f"[{i + 1}] {c}" for i, c in enumerate(candidates))
    response = llm.generate(
        LISTWISE_PROMPT.format(query=question, numbered_passages=numbered), provider=provider
    ).strip()
    match = re.search(r"<ranking>(.*?)</ranking>", response, re.S)

    order = []
    if match:
        for tok in match.group(1).replace(" ", "").split(","):
            if tok.isdigit():
                idx = int(tok) - 1
                if 0 <= idx < len(candidates) and idx not in order:
                    order.append(idx)
    for i in range(len(candidates)):
        if i not in order:
            order.append(i)

    return [candidates[i] for i in order[:top_n]]

Input → output, from an actual run — same scenario as cross-encoder reranking, for direct comparison:

Input question:
  What is the complexity per layer for a recurrent layer compared to
  self-attention?

The LLM's raw response (it reasons freely, then wraps only the final
answer in the requested tag):

  "Let me analyze each passage:

   Passage 4: This is key! Contains Table 1 showing:
   - Self-Attention: O(n²·d) complexity per layer
   - Recurrent: O(n·d²) complexity per layer
   This directly answers the question.

   Passage 1: Discusses sequential operations but mentions computational
   complexity, noting 'self-attention layers are faster than recurrent
   layers when the sequence...'
   [... more reasoning, several more passages, several more stray digits
   like 'Table 1' and '[4]' along the way ...]

   <ranking>4,1,9,3,5,11,2,6,13,14,7,10,15,8,12</ranking>"

Parsed ranking (from the tag only): passage 4 first — the correct
target — matching what cross-encoder reranking also found.

A prompting lesson this demo surfaced: 'return ONLY the list' doesn't work

The first version of this prompt instructed the model to return only a comma-separated list, nothing else. It didn't comply — it wrote several paragraphs of reasoning first, including plenty of stray digits ("Table 1", "[4]", "O(n²·d)") that have nothing to do with the actual ranking. A first attempt at parsing around this split the entire response on commas and grabbed the first few numeric-looking tokens — which pulled digits straight out of the reasoning prose instead of the real ranking, and silently returned the wrong top-3 with no error thrown anywhere.

The fix that actually holds up isn't a smarter parser fighting the model's behavior — it's a different prompt. Instead of demanding the model suppress reasoning it clearly wants to produce, the prompt above lets it reason freely and asks it to wrap only the final answer in an unambiguous <ranking>...</ranking> tag. Verified across several runs: 100% tag compliance, regardless of how much reasoning prose comes before it. Extraction is then one simple, unambiguous regex instead of a heuristic guessing which line is "the real answer." Don't fight a model's tendency to explain itself — give it an unambiguous place to put the part you actually need.

Measured: fixes the same miss, at real cost

On the identical scenario used for cross-encoder reranking, listwise reranking also correctly promotes the target chunk into the top-3. But the cost difference is stark and directly measured: scoring all 15 candidates with a pre-loaded cross-encoder took 0.036 seconds; the single listwise LLM call took 8.32 seconds — roughly 230x slower, for the same outcome on this scenario. That tracks with the broader research picture: LLM-based reranking can be modestly more accurate than cross-encoders in some settings, but at meaningfully higher latency and cost per query — cross-encoders reportedly get close to LLM-level accuracy at a fraction of the time.

Use it for: small candidate lists where you want the LLM reasoning about relevance holistically rather than scoring in isolation — useful when relevance genuinely depends on comparing candidates against each other, not just against the query alone. Skip it for: latency-sensitive paths, or any candidate set large enough that a cross-encoder gets you comparable results for a fraction of the cost — which, on the evidence here, is most of the time.

Disambiguation: RRF vs. Cross-Encoder Reranking

Easy to conflate — both involve "ranking," but they solve different problems at different stages.

RRF (Reciprocal Rank Fusion) — covered on the pre-retrieval page — merges several separate ranked lists (e.g. from multi-query paraphrases, or hybrid sparse+dense search) into one combined list. It scores purely on rank position — score = Σ 1/(k + rank) — no model inference, no re-reading of chunk or query, just arithmetic on ranks. Cheap and deterministic.

Cross-encoder reranking re-scores a single list of candidates by actually reading query and chunk together for a real relevance judgment — genuine model inference, not rank arithmetic.

They commonly run back-to-back, not as alternatives:

Query → [Multi-query generates 3 variants] → [3 separate retrievals]
      → [RRF merges the 3 ranked lists into 1 candidate list]
      → [Cross-encoder reranks that merged list for real relevance]
      → [Contextual compression trims the top-n] → [Final generation]

Scenario Check

Four scenarios testing whether you can match a post-retrieval symptom to the right technique.

Sources & further reading