Advanced RAG — Pre-retrieval Techniques

Naive RAG embeds the raw query, as-is, and searches. Every technique on this page intervenes before that search happens — rewriting, expanding, or reformulating the query itself, on the theory that a better-shaped query finds better chunks. That theory is correct, but only conditionally: each technique below fixes one specific, narrow failure mode, and does nothing (or actively adds noise) when that failure mode isn't present. The demos on this page are built around that idea directly — each technique is tested against a scenario that actually matches its own "when to use" case, so what you're seeing below is each technique doing the thing it was designed to do, not a technique judged against an unrelated question it was never meant to answer. The single most important skill on this page isn't memorizing which technique does what — it's recognizing which failure mode your own queries actually have, so you reach for the one technique that fixes it instead of applying all six as a blanket "best practice."

Hands-on

Every technique below is runnable, and every number is measured, not asserted: Query Transformation → in the code repo. All 6 need an LLM key — query transformation is inherently LLM-driven.

TL;DR — quick revision

• Query rewriting fixes a real miss on a genuinely vague, colloquial query — naive retrieval never finds the target passage; the rewritten, technically-phrased query reliably does. It has nothing to fix on an already-clear query, though, and blindly applying it everywhere just adds latency and occasional noise
• Multi-query/RAG-Fusion also fixes a real miss, on a broad, multi-faceted question where naive retrieval never surfaces one relevant aspect of the answer — but only in ~3 of 4 runs, since it depends on a fresh, unseeded LLM call generating paraphrases that happen to land near the target vocabulary. Real, but not guaranteed
• HyDE is unreliable in a specific, disclosed way: it mostly ties naive on numeric-fact questions (naive already does fine there), but can occasionally underperform when a safety-tuned LLM refuses to hallucinate a specific number instead of confidently fabricating one — which defeats the technique's actual mechanism
• Step-back prompting showed no measurable effect, even tested against a scenario built specifically to match its intended use case (a narrow fact with real general context nearby) — this single, well-organized paper doesn't separate general context from specific facts enough for the technique to add anything
• Query decomposition is a clean, real win: naive retrieval misses one of two facts in a compound question even at k=6; decomposition reliably finds both
• IRCoT (interleaved retrieval + reasoning) succeeds cleanly when the needed facts are each individually findable somewhere — but honestly reports "not found" rather than hallucinating when a fact is buried in a way no query phrasing can surface
• The overarching lesson: each technique fixes one specific failure mode. If your corpus and query distribution don't actually exhibit that failure mode, the technique adds cost for nothing — occasionally it actively hurts. Diagnose your actual failure mode first, then reach for the one technique built for it

Already covered elsewhere

Technique	What it does	See
Contextual Retrieval (Anthropic)	Prepend an LLM-generated context blurb to each chunk before embedding	Chunking Strategies
RAPTOR	Cluster chunks, summarize each cluster, search leaf + summary levels together	Retrieval Methods
Adaptive / FLARE-style retrieval	Only retrieve when the model's own confidence is low	Retrieval Methods

Query Rewriting

The problem it targets: users phrase queries badly — too short, too vague, colloquial, or using different words than the source document ever uses ("how do I get my money back" vs. a document that only ever says "refund"). Query rewriting passes the raw query through an LLM before retrieval, asking it to produce a clearer, more specific version — the rewritten query gets embedded and searched, not the original.

from sentence_transformers import SentenceTransformer

import llm

REWRITE_PROMPT = """Rewrite this query to be clearer and more specific \
for searching a technical document. Return ONLY the rewritten query, \
nothing else.

Query: {query}"""


def rewrite_query(query: str, provider: str | None = None) -> str:
    return llm.generate(REWRITE_PROMPT.format(query=query), provider=provider).strip()


def rewrite_retrieve(question: str, chunks: list[str], embed_model: SentenceTransformer, k: int = 3, provider: str | None = None) -> list[str]:
    import numpy as np

    def cos(a, b):
        return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b) + 1e-8))

    rewritten = rewrite_query(question, provider=provider)
    query_vec = embed_model.encode(rewritten, convert_to_numpy=True)
    chunk_vecs = embed_model.encode(chunks, convert_to_numpy=True, show_progress_bar=False)
    sims = [cos(query_vec, v) for v in chunk_vecs]
    top_indices = np.argsort(sims)[::-1][:k]
    return [chunks[i] for i in top_indices]

Input → output, from an actual run:

Input query:      why dont they just use RNNs like everyone else did before
Rewritten query:  What are the advantages of the proposed architecture over
                  traditional RNN-based approaches?

Naive top-1 chunk (wrong section):
  "...because it may allow the model to extrapolate to sequence lengths
  longer than the ones encountered during training. 4 Why Self-Attention
  In this section we comp..."

Rewrite top-1 chunk (right section):
  "...Recurrent models typically factor computation along the symbol
  positions of the input and output sequences. Aligning the positions to
  steps in computation time, they generate a sequence of hidden states ht...
  This inherently sequential nature precludes parallelization within
  training examples, which becomes critical at longer..."

Measured: fixes a real miss on a genuinely vague query

Asked the vague, colloquial question "why dont they just use RNNs like everyone else did before" — the kind of underspecified, non-technical phrasing rewriting exists to fix — naive retrieval never finds the passage explaining the model "precludes parallelization," at k=1, 2, or 3. Rewriting reliably finds it instead, at every k tested. (The exact rewritten wording varies run to run — it's a live LLM call — but it consistently lands on the model's technical vocabulary instead of the original colloquial phrasing.)

The mechanism is visible directly: the raw colloquial phrasing shares almost no vocabulary with the paper's own technical language ("precludes parallelization," "sequential computation"), so its embedding lands far from the answer. Rewriting bridges exactly that gap. But this only works because the query genuinely had a vocabulary-mismatch problem to begin with — rewrite an already-clear, well-formed question ("How many attention heads did they use?") and there's nothing to fix; the rewritten version is just a more formal restatement, and can even add noise (a rewrite might list plausible-but-wrong alternatives alongside the right answer, diluting the embedding's signal).

Where it has genuinely proven value — conversational rewriting: in multi-turn chat, a follow-up like "what about for enterprise?" is meaningless retrieved on its own — resolving it against conversation history into "What is the refund policy for enterprise customers?" is close to essential for any chatbot-style RAG; without it, retrieval on follow-ups silently fails, not just underperforms.

Use it for: vague/colloquial queries with real vocabulary mismatch (the case measured above), and conversational follow-ups (near-essential). Skip it for: queries that are already clear and specific relative to your corpus's own vocabulary — measure first, don't apply as a blanket "best practice."

Multi-Query Retrieval & RAG-Fusion

Different from rewriting — rewriting produces one improved query; multi-query generates several different phrasings of the same question, retrieves each independently, and merges the results. The theory: a single query embedding, however well-phrased, only points to one region of vector space — different phrasings can surface different relevant chunks a single query would miss.

Merging with RRF (Reciprocal Rank Fusion): each chunk's fused score is Σ 1/(k + rank), summed across every query variation's ranked list it appears in (k is a damping constant, typically 60). A chunk that ranks well across multiple phrasings outscores one that only a single phrasing liked — the intuition being that agreement across independent queries is a stronger relevance signal than one strong vote.

import numpy as np
from sentence_transformers import SentenceTransformer

import llm

MULTI_QUERY_PROMPT = """Generate 3 different phrasings of the question \
below, each capturing a different angle on the same underlying \
information need. Return one per line, nothing else.

Question: {query}"""


def reciprocal_rank_fusion(ranked_lists: list[list[str]], k: int = 60) -> list[str]:
    """Each chunk's fused score = sum of 1/(k + rank) across every list
    it appears in. A chunk ranked highly by multiple query variations
    outscores one that only one variation liked."""
    scores: dict[str, float] = {}
    for ranked in ranked_lists:
        for rank, chunk in enumerate(ranked):
            scores[chunk] = scores.get(chunk, 0.0) + 1.0 / (k + rank + 1)
    return sorted(scores, key=lambda c: -scores[c])


def multi_query_retrieve(question: str, chunks: list[str], embed_model: SentenceTransformer, k: int = 3, provider: str | None = None) -> list[str]:
    def cos(a, b):
        return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b) + 1e-8))

    response = llm.generate(MULTI_QUERY_PROMPT.format(query=question), provider=provider)
    variations = [q.strip("- ").strip() for q in response.split("\n") if q.strip()]
    variations.append(question)

    ranked_lists = []
    for variant in variations:
        query_vec = embed_model.encode(variant, convert_to_numpy=True)
        chunk_vecs = embed_model.encode(chunks, convert_to_numpy=True, show_progress_bar=False)
        sims = [cos(query_vec, v) for v in chunk_vecs]
        top_indices = np.argsort(sims)[::-1][:10]
        ranked_lists.append([chunks[i] for i in top_indices])

    return reciprocal_rank_fusion(ranked_lists)[:k]

Input → output, from an actual run where multi-query found what naive missed:

Input query:
  What lets this model connect words that are far apart in a sentence
  without losing track over long distances?

Generated paraphrases:
  1. What mechanism allows the model to maintain relationships between
     distant words across long sequences?
  2. How does the model preserve context when processing words that are
     separated by many tokens?
  3. What architectural feature enables the model to avoid degradation
     when linking words with large positional gaps between them?

Naive retrieval at k=5: none of the top 5 chunks mention path length at
all — the raw query's embedding never lands near that passage.

RRF-fused retrieval at k=5, rank 2 (right section, found via paraphrase #2):
  "...dependencies is a key challenge in many sequence transduction tasks.
  One key factor affecting the ability to learn such dependencies is the
  length of the paths forward and backward signals have to traverse in
  the network. The shorter these paths between any combination of
  positions in the input and output sequences, the easier it is to learn
  long-r[ange dependencies]..."

Measured: fixes a real miss on a broad question — most of the time

Asked "What lets this model connect words that are far apart in a sentence without losing track over long distances?" — a broad question whose answer could reasonably be phrased several different ways — naive retrieval never surfaces the passage explaining "maximum path length" (the mechanism that actually answers this) at k=5, across every run tested. Generating 3 paraphrases and fusing them with RRF found it in 3 of 4 runs.

That "3 of 4," not "4 of 4," is the honest number: multi-query makes a fresh, unseeded LLM call for its paraphrases every run, so the exact set of phrasings generated — and therefore whether one of them happens to land near the target passage's vocabulary — varies. This is real, disclosed non-determinism, not a bug; it means the fix is meaningfully more likely, not guaranteed. Note that this only helps because the question was genuinely broad and multi-faceted — on a narrow factual question with one unambiguous best-matching chunk, RRF has nothing to add: it can only dilute an already-optimal single match with weaker competing matches from the other paraphrases, never improve on it.

RAG-Fusion is the name for this specific combination — multi-query generation plus RRF-based merging — and it's a standard production pattern, but its value proposition is specifically "recall you'd otherwise miss with one query," not "better than the best single query on every question."

Use it for: ambiguous or multi-faceted questions where you have real evidence single-query retrieval misses relevant chunks. Skip it for: narrow factual lookups the corpus already answers cleanly with the raw query — the N extra retrieval calls (N=3-5 typically) buy nothing there.

HyDE (Hypothetical Document Embeddings)

The problem: a question is short and interrogative ("What optimizer did they use?"); the actual answer in the document reads as a statement ("We used the Adam optimizer..."). Embedding models sometimes struggle because a question's embedding isn't always closest in vector space to its answer's embedding, even when topically related. HyDE's fix: ask an LLM to write a plausible hypothetical answer — even if it's factually wrong — and embed that instead of the raw question. The hallucination is a scaffold, discarded after retrieval; only the real retrieved chunks make it into the final answer.

import numpy as np
from sentence_transformers import SentenceTransformer

import llm

HYDE_PROMPT = "Write a short passage that answers this question: {query}"


def hyde_retrieve(question: str, chunks: list[str], embed_model: SentenceTransformer, k: int = 3, provider: str | None = None) -> list[str]:
    hypothetical_answer = llm.generate(HYDE_PROMPT.format(query=question), provider=provider)
    query_vec = embed_model.encode(hypothetical_answer, convert_to_numpy=True)
    chunk_vecs = embed_model.encode(chunks, convert_to_numpy=True, show_progress_bar=False)
    sims = [float(np.dot(query_vec, v) / (np.linalg.norm(query_vec) * np.linalg.norm(v) + 1e-8)) for v in chunk_vecs]
    top_indices = np.argsort(sims)[::-1][:k]
    return [chunks[i] for i in top_indices]

Input → output, from an actual run:

Input question:
  What BLEU score did they get on English-to-German translation?

Hypothetical passage the LLM generated (this is what actually gets
embedded and searched — not the question above):
  "# BLEU Score for English-to-German Translation
   The neural machine translation system achieved a BLEU score of 27.8
   on the English-to-German translation task using the WMT 2014 dataset.
   This result represented a significant improvement over previous
   state-of-the-art systems at the time..."

Note the hallucinated number (27.8) is WRONG — the paper's real answer is
28.4. That doesn't matter for retrieval: HyDE only needs the passage to
be answer-SHAPED, not answer-correct. It gets discarded after retrieval;
only the real retrieved chunk feeds the final answer.

On a different run of the exact same question, the model instead refused to hallucinate:

"I don't have enough context to answer what BLEU score was achieved on
 English-to-German translation. To provide an accurate answer, I would
 need to know..."

Embedding that refusal text — which shares no real vocabulary with the paper's own reporting style — missed the target passage at k=1, where naive retrieval succeeded.

Measured: unreliable in a specific, disclosed way

Asked the numeric-fact question "What BLEU score did they get on English-to-German translation?", repeated across several runs: HyDE mostly tied naive retrieval — both find the "28.4" passage at k=1-3, since naive already does well here on the cleaned-up corpus. But in one run, asked to write a hypothetical answer to that same question, the LLM responded "I don't have enough context to answer this accurately..." instead of hallucinating a plausible number — and embedding that refusal text actually missed the target passage at k=1, where naive succeeded.

That's the real, underdiscussed tension worth flagging: HyDE's mechanism specifically wants a confident, answer-shaped hallucination as a retrieval scaffold — but well-aligned modern LLMs increasingly refuse to speculate on specific factual claims, which quietly undermines the technique exactly on the crisp numeric-fact questions it's often reached for. On a broader conceptual question instead — "how does the model track word order without recurrence?" — the LLM readily wrote a detailed hypothetical explanation every time, no refusal; but naive retrieval was already succeeding there too post-cleanup, so there was no real gap left for HyDE to close either way on this corpus.

Use it for: zero-shot/cross-domain retrieval where query style differs a lot from document style, and conceptual/explanatory questions the model will confidently answer rather than refuse. Skip it for: crisp numeric-fact questions, where a safety-tuned model may refuse to hallucinate a specific number — defeating the mechanism outright — or corpora where question and answer phrasing are already close, where HyDE adds an LLM call for a gap that isn't there.

Step-Back Prompting

Many questions are specific instances of a broader principle; retrieving narrowly for the specific question can miss the general background a reasoning-heavy answer actually needs. Step-back prompting asks an LLM to generate a more abstract "step-back" version of the query, retrieves for both the general and the specific question, and feeds both sets of context into generation — the only one of these techniques that deliberately broadens scope rather than narrowing or reformulating it.

import numpy as np
from sentence_transformers import SentenceTransformer

import llm

STEP_BACK_PROMPT = """What is a more general question that would help \
answer this specific question? Return ONLY the general question.

Specific question: {query}"""


def _naive_retrieve(question, chunks, embed_model, k):
    query_vec = embed_model.encode(question, convert_to_numpy=True)
    chunk_vecs = embed_model.encode(chunks, convert_to_numpy=True, show_progress_bar=False)
    sims = [float(np.dot(query_vec, v) / (np.linalg.norm(query_vec) * np.linalg.norm(v) + 1e-8)) for v in chunk_vecs]
    top_indices = np.argsort(sims)[::-1][:k]
    return [chunks[i] for i in top_indices]


def step_back_retrieve(question: str, chunks: list[str], embed_model: SentenceTransformer, k: int = 3, provider: str | None = None) -> list[str]:
    general_question = llm.generate(STEP_BACK_PROMPT.format(query=question), provider=provider).strip()

    specific_hits = _naive_retrieve(question, chunks, embed_model, k)
    general_hits = _naive_retrieve(general_question, chunks, embed_model, k)

    combined, seen = [], set()
    for chunk in specific_hits + general_hits:
        if chunk not in seen:
            combined.append(chunk)
            seen.add(chunk)
    return combined[:k]

Input → output, from an actual run:

Input question:         What value did they use for Pdrop during training?
Step-back question:     What hyperparameters were used during training?

Naive retrieval (specific question alone) finds "Pdrop = 0.1" at k=1.
Step-back's combined set (specific + general question, deduped) finds it
at k=1 too — the general question added nothing the specific one hadn't
already found.

Measured: no effect, even in a scenario built to fit its exact use case

Tested against a specific-fact-with-real-general-context question — "What value did they use for Pdrop during training?" (step-back generates the broader question "What hyperparameters were used during training?" for this one) — retrieval tied naive exactly at every k tested. The same held across several other candidate questions built the same deliberate way (a training-schedule question, an ablation-results question): naive, on its own, already found the specific fact directly every time, leaving nothing for the broader retrieval pass to add.

The likely reason: this page's sample document is a single, short, well-organized technical paper — general principles and specific facts sit close together throughout (a sentence about a hyperparameter and the paragraph explaining why it was chosen are typically adjacent). Step-back's real value case — think of a general policy rule and a specific year's numbers living in genuinely separate documents or sections far apart, like a company's tax brackets across separate annual filings — doesn't exist in a 15-page paper. A null result here says more about this corpus's structure than about the technique's real-world usefulness: it's a genuine negative result, not a sign the technique doesn't work.

Use it for: multi-step reasoning, comparative questions across time/context, or corpora where general principles and specific facts are genuinely far apart (long manuals, regulatory documents, anything spanning multiple sources). Skip it for: simple factual lookups, or any corpus small/coherent enough that context and fact already sit together.

Query Decomposition

Naive RAG embeds the whole query as one vector — if that query is actually a compound question bundling multiple distinct information needs, a single embedding represents a blurry average of all of them, and retrieval often surfaces chunks partially matching one part while missing the rest entirely. Decomposition detects this and splits the query into independent sub-questions, retrieving separately for each — different in kind from multi-query, which generates paraphrases of the same question rather than splitting into genuinely different ones.

import numpy as np
from sentence_transformers import SentenceTransformer

import llm

DECOMPOSE_PROMPT = """If the question below is asking multiple distinct \
things, break it into separate sub-questions, one per line. If it's \
already a single question, return it unchanged on one line.

Question: {query}"""


def decompose_retrieve(question: str, chunks: list[str], embed_model: SentenceTransformer, k: int = 6, provider: str | None = None) -> list[str]:
    """Larger k than a single-fact lookup needs — a compound question
    genuinely needs multiple slots, one set per sub-question, not one
    shared top-k the way a single-fact query would use."""

    def naive(q, kk):
        query_vec = embed_model.encode(q, convert_to_numpy=True)
        chunk_vecs = embed_model.encode(chunks, convert_to_numpy=True, show_progress_bar=False)
        sims = [float(np.dot(query_vec, v) / (np.linalg.norm(query_vec) * np.linalg.norm(v) + 1e-8)) for v in chunk_vecs]
        top_indices = np.argsort(sims)[::-1][:kk]
        return [chunks[i] for i in top_indices]

    response = llm.generate(DECOMPOSE_PROMPT.format(query=question), provider=provider)
    sub_questions = [q.strip("- ").strip() for q in response.split("\n") if q.strip()]

    per_question_k = max(1, k // len(sub_questions))
    combined, seen = [], set()
    for sub_q in sub_questions:
        for chunk in naive(sub_q, per_question_k):
            if chunk not in seen:
                combined.append(chunk)
                seen.add(chunk)
    return combined[:k]

Input → output, from an actual run:

Input question:
  What optimizer did they use, and how many attention heads did they use?

Sub-questions generated:
  1. What optimizer did they use?
  2. How many attention heads did they use?

Naive retrieval (single embedding, k=6): finds "h = 8" (attention heads),
never finds "Adam" (the optimizer) — the blurred embedding favors one
sub-topic and starves the other regardless of budget.

Decomposition (3 chunks retrieved per sub-question, k=6 total): finds
BOTH "Adam" and "h = 8" — each sub-question gets its own dedicated
retrieval pass instead of competing for the same embedding.

Measured: a clean, real win

Asked "What optimizer did they use, and how many attention heads did they use?" — a genuinely compound question — naive retrieval misses one of the two facts even at k=6: the single blurred embedding keeps favoring one sub-topic (attention heads) over the other (the optimizer) regardless of how much budget it's given. Decomposition, retrieving separately per sub-question, reliably surfaces both facts within that same k=6 budget. This is the one technique on this page where a single-fact question (the kind the rest of this page's questions are built from) couldn't demonstrate the real value case at all — decomposition correctly detects a non-compound question and leaves it alone, tying naive exactly — so a genuinely compound question was necessary to show what it's actually for.

Use it for: comparison questions, multi-part "and" questions, anything needing information from genuinely different sections the single embedding would never combine well. Skip it for: single-fact lookups — the decompose-detection call plus N separate retrieval passes cost real latency for a question that wasn't compound in the first place.

Multi-Hop Query Planning (IRCoT)

Decomposition splits a compound question upfront, all at once. IRCoT (Interleaved Retrieval with Chain-of-Thought) does something more dynamic: generate one reasoning step, use that step itself as the next retrieval query, retrieve, and repeat — each hop informed by what's actually been found so far, rather than a fixed plan made before any retrieval has happened. This targets genuinely multi-hop questions where the second lookup depends on what the first one turns up, not just "two unrelated facts asked in one sentence."

import numpy as np
from sentence_transformers import SentenceTransformer

import llm

IRCOT_REASON_PROMPT = """You are answering a question using a document \
retrieval system — you do NOT have the document memorized, but each \
reasoning step you write will be used as a search query against it.

Question: {question}

Retrieved so far:
{context}

Write ONE short, specific, searchable sentence describing the next \
fact you need to look up to make progress (this sentence becomes the \
next search query, so keep it concrete and on-topic — do not ask what \
document this is, assume it's a technical ML paper). If the retrieved \
information above is already enough to answer, start your sentence \
with "ANSWER:" instead."""

FINAL_SYNTHESIS_PROMPT = """Question: {question}

Retrieved passages across {n_hops} rounds of search:
{context}

Using ONLY the passages above, give the final answer in 1-2 sentences. \
If they genuinely don't contain enough to answer, say so plainly."""


def ircot_retrieve(question: str, chunks: list[str], embed_model: SentenceTransformer, k: int = 3, provider: str | None = None, max_hops: int = 3) -> dict:
    """Interleave reasoning and retrieval: generate one reasoning step,
    use THAT step as the next retrieval query, and repeat — each hop's
    retrieval is informed by the REAL retrieved text seen so far (not
    just the model's own prior reasoning sentences — that distinction
    matters: without real evidence to look at, the model has no way to
    judge whether it actually knows enough yet)."""

    def naive(q, kk):
        query_vec = embed_model.encode(q, convert_to_numpy=True)
        chunk_vecs = embed_model.encode(chunks, convert_to_numpy=True, show_progress_bar=False)
        sims = [float(np.dot(query_vec, v) / (np.linalg.norm(query_vec) * np.linalg.norm(v) + 1e-8)) for v in chunk_vecs]
        top_indices = np.argsort(sims)[::-1][:kk]
        return [chunks[i] for i in top_indices]

    context_chunks: list[str] = []
    reasoning_trace: list[str] = []

    for _ in range(max_hops):
        context_text = "\n---\n".join(context_chunks) if context_chunks else "(nothing retrieved yet)"
        step = llm.generate(IRCOT_REASON_PROMPT.format(question=question, context=context_text), provider=provider).strip()
        reasoning_trace.append(step)

        if step.upper().startswith("ANSWER:"):
            break

        for chunk in naive(step, kk=2):
            if chunk not in context_chunks:
                context_chunks.append(chunk)

    final_context = "\n---\n".join(context_chunks) if context_chunks else "(nothing retrieved)"
    final_answer = llm.generate(
        FINAL_SYNTHESIS_PROMPT.format(question=question, n_hops=len(reasoning_trace), context=final_context),
        provider=provider,
    )

    return {"reasoning_trace": reasoning_trace, "retrieved_chunks": context_chunks[:k], "final_answer": final_answer}

Getting this right took two real fixes, not just a clean write-up: the first version only showed the model its own prior reasoning sentences at each hop, never the actual retrieved chunk text — with no real evidence to look at, the model had no way to judge whether it actually knew enough, and just kept generating near-identical restated queries forever. The fix was mechanical but essential: feed the real retrieved passages back into context at every hop, and add a forced final-synthesis call at the end so the loop always commits to a real answer instead of trailing off in a hedge.

A real success — sequential hops correctly identifying what's still missing

Asked "What optimizer did they use, and how many attention heads did they use?", one run went: Hop 1 — "What optimizer was used in the model training?" (finds Adam). Hop 2 — "I need to find information about the number of attention heads" (finds h=8). Hop 3 — "ANSWER: They used the Adam optimizer... and h=8 parallel attention heads." Final synthesis correctly combined both, fully grounded. Each hop's query was shaped by what the previous hop had (and hadn't) turned up — genuinely adaptive, not a fixed script.

A real failure — and it isn't IRCoT's fault

Asked to find which company an author (credited in an informally-written footnote) is affiliated with, IRCoT failed after 3 hops of near-identical restated queries, and — correctly — reported it couldn't find the answer rather than guessing. Checking why: a single, perfectly-phrased direct query for that exact fact also fails to find it in the top-5, with no query transformation involved at all. The footnote text embeds poorly with a general-purpose dense model regardless of how the query is phrased. No amount of reasoning-and-retrieval interleaving can chain together a step that individually isn't findable — this is the same lesson FLARE-style adaptive retrieval already surfaced: query strategy controls how and when you search, not what the underlying embedding model is fundamentally capable of representing.

Use it for: genuinely multi-hop questions where a later lookup depends on an earlier one's answer, and where the individual facts involved are realistically findable by some phrasing. Skip it for: questions that are really just several independent facts (decomposition is cheaper and equally effective there), or corpora where the specific fact needed is known to be poorly represented by the embedding model regardless of query strategy — no amount of reasoning fixes that ceiling.

The pattern across all six

Technique	Measured effect, on a scenario matching its use case	Real value case
Query rewriting	Fixes a real miss (vague query)	Vague/colloquial queries, conversational follow-ups
Multi-query/RAG-Fusion	Fixes a real miss, in ~3 of 4 runs	Ambiguous queries where single-query retrieval demonstrably misses chunks
HyDE	Unreliable — ties normally, can hurt on numeric facts	Cross-domain retrieval, question/answer phrasing mismatch
Step-back prompting	No effect, even scenario-matched	Reasoning across genuinely separated general/specific content
Query decomposition	Clean win (on a compound question)	Genuinely compound, multi-part questions
IRCoT	Succeeds when facts are individually findable	Multi-hop questions with real sequential dependency

Two clean wins, one technique whose unreliability is itself the honest finding, one genuine null result, and one technique whose success is entirely conditional on the underlying fact being findable at all — out of six techniques that are all, individually, real and well-established. The thread connecting all six: every one of them targets a specific failure mode in how a query relates to your corpus. Before reaching for any of them, ask which failure mode your own queries actually have — vague phrasing, a broad multi-faceted question, a question/answer style mismatch, missing general context, a compound question, or a genuinely multi-hop one. Apply the technique that matches; skip the rest. The same discipline applies to reranking and chunking strategy elsewhere on this site — measure against your own data before adding any technique to a pipeline.

Scenario Check

Six scenarios testing whether you can match a query symptom to the right technique — and whether you know when a technique's absence of effect is itself informative.

Sources & further reading

• RAG-Fusion: a New Take on Retrieval-Augmented Generation — multi-query + RRF
• Precise Zero-Shot Dense Retrieval without Relevance Labels — the original HyDE paper
• Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models — the original step-back prompting paper
• Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions — the original IRCoT paper