Retrieval Methods

Chunking decides what units exist to retrieve; the embedding model decides how "similar" gets measured. This page is about everything left over — the actual retrieval-time decisions that determine which of those units come back, in what order, and how much of the index gets touched. Six methods, each demonstrated with a real, run result rather than a description of what it's supposed to do.

Hands-on

Every technique below is runnable: Retrieval Methods → in the code repo. Two methods run fully locally; four need an LLM key.

TL;DR — quick revision

• MMR (Maximal Marginal Relevance) diversifies top-k, trading some pure relevance for less redundancy — a real, tested langchain_core utility, not something to hand-roll
• Self-query retrieval has an LLM split a query into a semantic part + structured metadata filters — but only works if the LLM is told the valid values for each field, not just the field names (measured: silently dropped a filter until the schema was explicit)
• Hypothetical Questions generates plausible questions per chunk at index time, so query-time comparison is question-to-question, not question-to-answer — the mirror image of HyDE, which instead generates a hypothetical answer at query time
• Hierarchical index retrieval narrows document → section → chunk in stages, instead of searching every chunk flat
• RAPTOR clusters chunks and summarizes each cluster, so broad synthesis questions can match a summary node instead of failing to match any single leaf chunk
• Adaptive retrieval (FLARE-style) only retrieves when the model's own self-reported confidence is low — measured: a too-permissive threshold let a wrong FLOPS figure ship ungrounded; a stricter one caught it, though even then, single-vector retrieval alone didn't find the actual answer

Already covered elsewhere

A few retrieval-adjacent techniques got full treatment on other pages already — cross-referenced here instead of repeated:

Technique	What it does	See
Sparse (BM25), dense, learned sparse (SPLADE), hybrid + RRF fusion	The three retrieval paradigms, measured head-to-head on the same corpus, and how to combine them	Embedding Model Selection → BM25 vs. dense vs. SPLADE
Query rewriting, multi-query/RAG-Fusion, HyDE, step-back, decomposition	Fixing query quality before retrieval runs	Advanced RAG — Pre-retrieval
Cross-encoder reranking, contextual compression	Fixing result quality after retrieval runs	Advanced RAG — Post-retrieval
Small-to-big / parent-document retrieval	Precise child-chunk matching, full-parent-chunk generation	Chunking Strategies → Small-to-Big / Parent-Document Retrieval
Graph-based retrieval (traversal)	Walking an entity/relationship graph instead of vector similarity	Graph RAG → Querying the graph
Cross-modal retrieval (search-both-and-fuse)	Searching per-modality indexes, fusing with RRF	Multimodal RAG → Separate indexes + RRF fusion

Maximal Marginal Relevance (MMR)

Plain top-k retrieval has no concept of redundancy: if five near-duplicate chunks all happen to be the closest matches to a query, plain top-k happily returns all five, wasting four of your k slots on saying the same thing over and over while a genuinely different, also-relevant chunk gets pushed out. MMR fixes this by scoring each candidate on both relevance to the query and dissimilarity to whatever's already been selected, picking greedily: relevance minus redundancy, one slot at a time.

Parameters: lambda_mult — 1.0 is pure relevance (identical to plain top-k), 0.0 is pure diversity (ignores the query almost entirely once the first pick is made). 0.5-0.7 is a reasonable starting point; tune based on how much redundancy your actual corpus tends to produce.

Measured, using the real langchain_core.vectorstores.utils.maximal_marginal_relevance utility — not a hand-rolled reimplementation, since this is exactly the kind of well-established algorithm worth using a tested implementation for:

import numpy as np
from langchain_core.vectorstores.utils import maximal_marginal_relevance
from sentence_transformers import SentenceTransformer


def mmr_retrieve(
    query: str, chunks: list[str], embed_model: SentenceTransformer, k: int = 5, lambda_mult: float = 0.5
) -> list[str]:
    """Diversify top-k results: greedily pick the chunk that's most
    relevant to the query MINUS most similar to chunks already picked.
    `lambda_mult=1.0` is pure relevance (identical to plain top-k);
    `lambda_mult=0.0` is pure diversity, ignoring the query entirely."""
    query_vec = embed_model.encode(query, convert_to_numpy=True)
    chunk_vecs = embed_model.encode(chunks, convert_to_numpy=True, show_progress_bar=False)
    selected_indices = maximal_marginal_relevance(query_vec, list(chunk_vecs), lambda_mult=lambda_mult, k=k)
    return [chunks[i] for i in selected_indices]

At lambda_mult=1.0 on the sample document, every one of the top-4 results is about multi-head attention specifically — relevant, but repetitive. At lambda_mult=0.3, the result set pulls in a training-cost footnote and an author-contributions passage alongside the attention content — genuinely different information, at some cost to pure topical focus.

Use it for: result sets prone to near-duplication — FAQ-style corpora with several similarly-worded entries, or any corpus where the top-k from plain similarity search tends to cluster tightly around one narrow restatement of the same point. Skip it for: corpora where the top matches are already naturally diverse, or latency-sensitive paths where the extra pairwise-similarity computation between candidates isn't worth the cost.

Self-Query Retrieval

Most retrieval treats a query as pure semantic content. Self-query retrieval recognizes that a lot of real queries actually contain two different kinds of information mixed together — "what's the 2024 remote work policy" is a semantic search for "remote work policy" plus a hard filter, year = 2024. An LLM splits the query into these two parts; the filter gets applied first (a hard constraint, not a similarity signal), and only the surviving candidates get ranked by semantic similarity.

import json

from sentence_transformers import SentenceTransformer


def _cosine_sim(a, b):
    import numpy as np

    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b) + 1e-8))


SELF_QUERY_PROMPT = """Given the natural-language query below, extract:
1. "semantic_query": the part that should be matched by meaning
2. "filters": any metadata constraints mentioned, as a JSON object with
   only the keys below — omit any key not clearly implied by the query.

Available metadata fields (this is the schema — without it, the model
has no way to know these are valid categorical values, not just query
content; this is exactly what a real self-query retriever's field-info
schema provides):
- "section": one of "remote_work", "expenses", "benefits"
- "year": an integer, e.g. 2023 or 2024

Return ONLY a JSON object like {{"semantic_query": "...", "filters": {{...}}}}.

Query: {query}"""


def self_query_retrieve(query: str, corpus: list[dict], generate_fn, k: int = 3) -> list[dict]:
    """Ask an LLM to split a natural-language query into a semantic part
    and structured metadata filters, then apply the filters BEFORE
    ranking by semantic similarity. `generate_fn` is any `str -> str`
    function that calls an LLM (e.g. an Anthropic/OpenAI client call) —
    swap in whichever provider you use."""
    response = generate_fn(SELF_QUERY_PROMPT.format(query=query))
    parsed = json.loads(response.strip().strip("`").removeprefix("json").strip())
    semantic_query = parsed["semantic_query"]
    filters = parsed.get("filters", {})

    candidates = [doc for doc in corpus if all(doc.get(key) == value for key, value in filters.items())]
    if not candidates:
        return []

    embed_model = SentenceTransformer("all-MiniLM-L6-v2")
    query_vec = embed_model.encode(semantic_query, convert_to_numpy=True)
    doc_vecs = embed_model.encode([doc["text"] for doc in candidates], convert_to_numpy=True, show_progress_bar=False)
    scored = sorted(zip(candidates, doc_vecs), key=lambda pair: -_cosine_sim(query_vec, pair[1]))
    return [doc for doc, _ in scored[:k]]

A real failure this exact prompt produced, and the fix

The first version of this prompt told the LLM the metadata fields existed ("section", "year") but not what values "section" could actually take. Given the query "What's the 2024 remote work policy?", the model extracted only {"year": 2024} — it treated "remote work" purely as semantic content, never recognizing it as a valid value for a categorical field it didn't know existed. The retrieved results correctly matched year=2024 but silently included an unrelated expenses entry alongside the two real remote_work matches.

The fix: tell the model the field's actual valid values — "section": one of "remote_work", "expenses", "benefits". With that added, the same query correctly extracts both {"section": "remote_work", "year": 2024}, and the result set narrows to exactly the two matching entries. This is precisely why LangChain's real SelfQueryRetriever requires an explicit AttributeInfo schema per field, not just field names — a schema-free prompt has no way to know which words in a query are meant as filters versus just topic content.

Use it for: knowledge bases with real, structured metadata a user might naturally reference in plain language — dates, categories, authors, statuses — where hand-building a filter UI is more friction than letting the LLM infer it. Skip it for: unstructured corpora with no meaningful metadata to filter on, or metadata schemas complex enough that reliable extraction needs more than a short field-value schema (at that point, a structured search form is more reliable than hoping the LLM infers correctly every time).

Hypothetical Questions

HyDE fixes the question-vs-answer phrasing gap at query time, by generating a hypothetical answer and embedding that instead of the raw question. Hypothetical Questions fixes the exact same gap from the opposite direction, at index time: for each chunk, an LLM generates several plausible questions that chunk could answer, and those generated questions — not the chunk's own declarative text — get embedded and indexed. At query time, the user's real question is compared against other questions, not against answer-shaped text — same "shape" of text on both sides of the comparison.

from sentence_transformers import SentenceTransformer


def _cos(a, b):
    import numpy as np

    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b) + 1e-8))


HYPOTHETICAL_QUESTIONS_PROMPT = """Generate 3 short questions that the \
passage below could answer. Return one question per line, nothing else.

Passage: {passage}"""


def build_hypothetical_questions_index(chunks: list[str], generate_fn) -> list[dict]:
    """At INDEX time (not query time), generate plausible questions each
    chunk could answer, and embed those questions instead of the chunk's
    own declarative text — question-to-question similarity at query time
    is often stronger than question-to-answer similarity."""
    index = []
    for chunk in chunks:
        response = generate_fn(HYPOTHETICAL_QUESTIONS_PROMPT.format(passage=chunk))
        questions = [q.strip("- ").strip() for q in response.split("\n") if q.strip()]
        for question in questions:
            index.append({"question": question, "chunk": chunk})
    return index


def retrieve_via_hypothetical_questions(
    query: str, question_index: list[dict], embed_model: SentenceTransformer, k: int = 3
) -> list[str]:
    query_vec = embed_model.encode(query, convert_to_numpy=True)
    question_vecs = embed_model.encode(
        [item["question"] for item in question_index], convert_to_numpy=True, show_progress_bar=False
    )
    scored = sorted(zip(question_index, question_vecs), key=lambda pair: -_cos(query_vec, pair[1]))
    seen_chunks, results = set(), []
    for item, _ in scored:
        if item["chunk"] not in seen_chunks:
            results.append(item["chunk"])
            seen_chunks.add(item["chunk"])
        if len(results) >= k:
            break
    return results

Trade-off vs. HyDE: Hypothetical Questions pays its LLM cost once, upfront, at indexing time — query time is a normal, fast embedding lookup with no extra LLM call. HyDE pays an LLM call on every single query, with zero indexing overhead. Pick Hypothetical Questions when query volume is high relative to how often the corpus changes; pick HyDE when the corpus changes too often to justify re-generating questions for every chunk repeatedly.

Use it for: high query-volume systems built on a relatively stable knowledge base. Skip it for: a corpus that changes constantly, where the indexing cost would need to be paid over and over.

Hierarchical Index Retrieval

Searching every chunk in a large corpus flat gets slower and noisier as the corpus grows — more opportunities for an irrelevant chunk from an unrelated document to score deceptively high purely by coincidence. Hierarchical index retrieval builds multiple independently-searchable levels — document-level, section-level, chunk-level — and searches them in stages: find the right document first, then the right section within it, then the right chunk within that section. The same instinct that makes a person go to the right book, then the right chapter, then the right paragraph, rather than scanning every page of every book in a library.

import numpy as np
from sentence_transformers import SentenceTransformer


def hierarchical_retrieve(
    query: str, chunks: list[str], embed_model: SentenceTransformer, section_size: int = 5, k: int = 3
) -> list[str]:
    """Narrow the search space in two stages instead of searching every
    chunk flat: group chunks into sections, embed each section (here, by
    embedding its concatenated text — production systems often use an
    LLM summary instead for a sharper section-level representation),
    find the best-matching section first, then rank only its chunks."""

    def cos(a, b):
        return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b) + 1e-8))

    sections = [chunks[i : i + section_size] for i in range(0, len(chunks), section_size)]
    section_texts = [" ".join(section) for section in sections]

    query_vec = embed_model.encode(query, convert_to_numpy=True)
    section_vecs = embed_model.encode(section_texts, convert_to_numpy=True, show_progress_bar=False)
    best_section_idx = int(np.argmax([cos(query_vec, vec) for vec in section_vecs]))

    candidate_chunks = sections[best_section_idx]
    chunk_vecs = embed_model.encode(candidate_chunks, convert_to_numpy=True, show_progress_bar=False)
    scored = sorted(zip(candidate_chunks, chunk_vecs), key=lambda pair: -cos(query_vec, pair[1]))
    return [chunk for chunk, _ in scored[:k]]

How this differs from small-to-big/parent-document (easy to conflate — both involve parent-child structure): small-to-big is a 2-tier system built for one purpose — retrieve precisely on a small child chunk, then fetch a larger parent purely for generation context; only the child tier is ever searched. Hierarchical index retrieval is typically 3+ tiers, and every tier is embedded and independently searchable — the point is narrowing the search space itself at each stage, not just fetching more context after a match is already found.

Use it for: large-scale, many-document corpora where a flat search is measurably slow or noisy. Skip it for: a single small document's internal retrieval, where there's no meaningful "which document" stage to narrow first.

RAPTOR

Flat retrieval — even hybrid, even reranked — struggles with questions that need synthesis across many parts of a document, not a single fact lookup: "what is this paper about overall" has no single chunk that answers it. RAPTOR builds a tree to fix this: cluster chunks by embedding similarity, summarize each cluster with an LLM into a higher-level node, then recurse — cluster the summaries themselves into a higher layer, summarize again — up toward a small number of broad summary nodes. At query time, search leaf chunks and summary nodes together: a narrow factual question tends to match best at leaf level; a broad synthesis question tends to match a summary node instead.

import numpy as np
from sentence_transformers import SentenceTransformer
from sklearn.cluster import KMeans


def _cosine(a, b):
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b) + 1e-8))


RAPTOR_SUMMARY_PROMPT = """Summarize the following related passages in \
2-3 sentences, capturing the common theme:

{passages}"""


def build_raptor_tree(
    chunks: list[str], embed_model: SentenceTransformer, generate_fn, n_clusters: int = 3
) -> dict:
    """Recursively cluster chunks by embedding similarity and summarize
    each cluster with an LLM, building one extra tree layer. (The full
    RAPTOR paper recurses this multiple times up to a single root
    summary; one layer is enough to demonstrate the mechanism without
    the full offline cost.)"""
    chunk_vecs = embed_model.encode(chunks, convert_to_numpy=True, show_progress_bar=False)
    n_clusters = min(n_clusters, len(chunks))
    labels = KMeans(n_clusters=n_clusters, n_init=10, random_state=0).fit_predict(chunk_vecs)

    summary_nodes = []
    for cluster_id in range(n_clusters):
        cluster_chunks = [c for c, label in zip(chunks, labels) if label == cluster_id]
        if not cluster_chunks:
            continue
        summary = generate_fn(RAPTOR_SUMMARY_PROMPT.format(passages="\n\n".join(cluster_chunks)))
        summary_nodes.append({"summary": summary, "cluster_chunks": cluster_chunks})

    return {"leaves": chunks, "summaries": summary_nodes}


def raptor_retrieve(query: str, tree: dict, embed_model: SentenceTransformer, k: int = 2) -> list[str]:
    """Search leaf chunks AND summary nodes together — a narrow factual
    question tends to match best at leaf level; a broad synthesis
    question tends to match best at a summary node."""
    query_vec = embed_model.encode(query, convert_to_numpy=True)

    candidates = [(leaf, "leaf") for leaf in tree["leaves"]]
    candidates += [(node["summary"], "summary") for node in tree["summaries"]]

    texts = [c[0] for c in candidates]
    vecs = embed_model.encode(texts, convert_to_numpy=True, show_progress_bar=False)
    scored = sorted(zip(candidates, vecs), key=lambda pair: -_cosine(query_vec, pair[1]))
    return [f"[{kind}] {text}" for (text, kind), _ in scored[:k]]

Run against the sample paper, real clustering (scikit-learn's KMeans) plus real LLM summarization produced genuinely coherent cluster summaries — one describing the shift away from recurrent networks toward pure attention, another describing the paper's authorship and permissions, another describing the BLEU-score results and training efficiency. Asked the broad question "What is this paper about overall?", retrieval correctly surfaced a summary node rather than a single narrow leaf chunk that could only ever answer one piece of that question.

Trade-off: real offline cost — clustering plus an LLM summarization call per cluster, at every tree layer, done once at indexing time (or whenever the corpus changes). Query-time cost stays close to normal vector search, just against a richer, multi-level index.

Use it for: long documents or corpora where "what's the overall theme" questions are common, not just point lookups. Skip it for: corpora that are almost entirely single-fact lookups, where the extra indexing cost buys nothing.

Adaptive Retrieval (a FLARE-style approach)

Instead of always retrieving a fixed top-k for every query regardless of whether it's actually needed, adaptive retrieval decides dynamically whether to retrieve at all. The real mechanism behind this in the research literature is FLARE (Forward-Looking Active REtrieval): monitor the model's own token-level confidence during generation, and only trigger a retrieval call when confidence drops on a specific claim — using the low-confidence sentence itself as the retrieval query.

Real token-level log-probabilities aren't exposed the same way across every hosted chat API, so the runnable version here uses a simpler, fully implementable stand-in: generate a draft answer, then ask the model to self-report its own confidence in that answer (1-5), and only retrieve if the self-reported score is low. Cruder than real FLARE's live token-probability signal, but the same "retrieve only when needed" shape, and it works against any chat-completions API.

import re

import numpy as np
from sentence_transformers import SentenceTransformer

DRAFT_PROMPT = "Answer this question in one sentence, from general knowledge: {question}"
CONFIDENCE_PROMPT = """Sentence: "{sentence}"

On a scale of 1 (pure guess) to 5 (certain, well-known fact), how \
confident are you this sentence is factually correct? Reply with ONLY \
the number."""


def adaptive_retrieve(
    question: str,
    chunks: list[str],
    embed_model: SentenceTransformer,
    generate_fn,
    confidence_threshold: int = 4,
) -> dict:
    """A simplified stand-in for FLARE: instead of monitoring token-level
    log-probabilities during generation (which needs logprob access most
    hosted chat APIs don't expose the same way), ask the model to
    self-report confidence in its own draft answer, and only retrieve
    (and regenerate) if that self-reported confidence is low. Cruder
    than real FLARE, but the same "retrieve only when needed" shape."""
    draft = generate_fn(DRAFT_PROMPT.format(question=question))
    confidence_response = generate_fn(CONFIDENCE_PROMPT.format(sentence=draft))
    confidence = int(re.search(r"\d", confidence_response).group())

    if confidence >= confidence_threshold:
        return {"answer": draft, "confidence": confidence, "retrieved": False}

    def cos(a, b):
        return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b) + 1e-8))

    query_vec = embed_model.encode(question, convert_to_numpy=True)
    chunk_vecs = embed_model.encode(chunks, convert_to_numpy=True, show_progress_bar=False)
    best_idx = int(np.argmax([cos(query_vec, vec) for vec in chunk_vecs]))
    context = chunks[best_idx]

    grounded_prompt = f"Using ONLY this context, answer in one sentence: {context}\n\nQuestion: {question}"
    grounded_answer = generate_fn(grounded_prompt)
    return {"answer": grounded_answer, "confidence": confidence, "retrieved": True, "context_used": context}

A real failure, and what fixing it did and didn't solve

Asked "What is the exact FLOPS count for training the big Transformer model?" with confidence_threshold=3, the model self-reported confidence 3, cleared the threshold, and confidently answered 3.3×10¹⁸ FLOPs. The actual paper reports 3.3×10¹⁸ for the base model and 2.3×10¹⁹ for the big model — a full order of magnitude apart. Confidence 3 was exactly high enough to skip retrieval and exactly low enough that the claim was, in fact, wrong.

Raising the threshold to 4 fixed the triggering problem: the same low confidence now correctly triggers a retrieval pass instead of shipping the answer directly. But it didn't fully fix the answer — plain top-1 dense retrieval on the sample document doesn't surface the actual FLOPS table row at all (it doesn't even rank in the top 5), because a numeric table row ("Transformer (big) 28.4 41.8 2.3· 10¹⁹") embeds poorly with a general-purpose dense model — exactly the dense-embeddings blind spot from the embedding model selection page. The system correctly said "not found in this context" rather than hallucinating a number — a real win — but adaptive retrieval only controls whether to retrieve; it can't fix what the underlying retrieval method is actually capable of finding.

Use it for: high-stakes factual QA where an ungrounded wrong answer is costly, and where retrieval cost per query genuinely needs to scale with question difficulty rather than being paid uniformly on every call. Skip it for: high-throughput, low-latency use cases where the extra self-assessment call isn't worth the cost, or where self-reported LLM confidence hasn't been validated to actually correlate with correctness on your own data — as this exact result shows, it doesn't always.

Scenario Check

Six scenarios testing whether you can match a retrieval symptom to the right method.

Sources & further reading

• MMR — Maximal Marginal Relevance for search result diversification — Elastic Search Labs
• LangChain: Self-querying retrievers — the AttributeInfo schema pattern
• RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval — the original paper
• Active Retrieval Augmented Generation (FLARE) — the original paper introducing token-confidence-triggered retrieval