Revision

Every page's TL;DR in one place, every page's Scenario Check merged into one combined quiz, and a flashcard deck for fast recall drilling — for a quick pass before an interview instead of clicking through pages one at a time.

Kept in sync manually

This page is generated from the TL;DR and Scenario Check blocks on every content page by scripts/generate_revision.py, re-run after editing any page's TL;DR or quiz — it isn't live-generated on every build.

All TL;DRsCombined Scenario CheckFlashcards

Foundations

Naive RAG

• Pipeline: chunk → embed → store (offline, once) · embed query → similarity search → top-k → prompt → generate (online, per query)
• "Naive" = one retrieve-then-generate pass, no loops, no reranking, no self-checks
• Five failure modes: semantic similarity ≠ relevance · no query understanding · fixed top-k with no reranking · no groundedness check · can't handle multi-part questions
• Hard rule: query and chunks must be embedded with the exact same embedding model, or similarity scores become meaningless
• Use it when: small, clean, well-scoped KB, low-stakes lookups, prototyping
• Upgrade path: Advanced RAG (pre/post-retrieval) is the cheapest next step for almost every one of these failure modes

Advanced RAG — Pre-retrieval

• Query rewriting fixes a real miss on a genuinely vague, colloquial query — naive retrieval never finds the target passage; the rewritten, technically-phrased query reliably does. It has nothing to fix on an already-clear query, though, and blindly applying it everywhere just adds latency and occasional noise
• Multi-query/RAG-Fusion also fixes a real miss, on a broad, multi-faceted question where naive retrieval never surfaces one relevant aspect of the answer — but only in ~3 of 4 runs, since it depends on a fresh, unseeded LLM call generating paraphrases that happen to land near the target vocabulary. Real, but not guaranteed
• HyDE is unreliable in a specific, disclosed way: it mostly ties naive on numeric-fact questions (naive already does fine there), but can occasionally underperform when a safety-tuned LLM refuses to hallucinate a specific number instead of confidently fabricating one — which defeats the technique's actual mechanism
• Step-back prompting showed no measurable effect, even tested against a scenario built specifically to match its intended use case (a narrow fact with real general context nearby) — this single, well-organized paper doesn't separate general context from specific facts enough for the technique to add anything
• Query decomposition is a clean, real win: naive retrieval misses one of two facts in a compound question even at k=6; decomposition reliably finds both
• IRCoT (interleaved retrieval + reasoning) succeeds cleanly when the needed facts are each individually findable somewhere — but honestly reports "not found" rather than hallucinating when a fact is buried in a way no query phrasing can surface
• The overarching lesson: each technique fixes one specific failure mode. If your corpus and query distribution don't actually exhibit that failure mode, the technique adds cost for nothing — occasionally it actively hurts. Diagnose your actual failure mode first, then reach for the one technique built for it

Advanced RAG — Post-retrieval

• Cross-encoder reranking fixes a real miss: naive bi-encoder retrieval finds the right chunk somewhere in a wider candidate pool but ranks it outside the top-3; reranking with a cross-encoder correctly promotes it — because it reads query and chunk together, instead of comparing two independently-computed vectors
• Contextual compression cuts ~94% of the retrieved text on a typical multi-chunk retrieval, keeping only the sentences that actually answer the question — same underlying fact, far less padding sent to the final LLM call
• LLM-based listwise reranking (RankGPT-style) fixes the same miss cross-encoder reranking does — but at roughly 230x the latency (one full LLM round-trip vs. one local forward pass), since it sends the whole candidate list to the LLM in a single call instead of scoring each candidate independently
• A real prompting lesson worth knowing: telling the model to "return ONLY the list" doesn't work — it explains its reasoning anyway. Fighting that after the fact (parsing around the prose) is fragile; asking it to wrap just the final answer in an unambiguous <ranking>...</ranking> tag and letting it reason freely everywhere else is the fix that actually holds up
• The overarching lesson: reranking's value depends on naive retrieval actually having a ranking problem — recall the chunk, rank it wrong. On a small, well-organized corpus, naive bi-encoder retrieval is often already good enough that there's nothing to fix; reranking earns its keep on messier, larger corpora with more distractor chunks

Architectures

Modular RAG

• The same routing decision, built 3 ways: keyword heuristics (no model call), embedding-based semantic routing (one local embedding call, no LLM), and the exact LLM router already built in adaptive-rag/
• Measured on two separate sets, deliberately: a "design set" (the queries the heuristic rules were written looking at) and a genuinely held-out set (written without looking at those rules) — because a router's accuracy on the queries it was tuned against tells you almost nothing about how it'll do on the next query someone actually asks
• The heuristic router's accuracy collapses from 1.00 to 0.40 moving from the design set to held-out phrasing — real, measured brittleness, not a hypothetical caveat about rule-based systems
• The embedding-based router is the more honest number: 0.67 → 0.80 across the same two sets — roughly consistent, because it was never fit to either one
• The LLM router is the only one both accurate and consistent everywhere — 1.00 on both sets — at roughly 30-1500x the latency of the other two, and a real per-call cost they don't have

Self-RAG

• Two implementation paths exist, and they're easy to conflate: the original paper fine-tunes a model to emit special reflection tokens; almost everyone in production instead uses a prompted version — a pipeline of separate LLM calls doing the same grading jobs, no fine-tuning at all. Both are legitimately "Self-RAG."
• The prompted loop, in order: retrieve → grade each chunk's relevance → generate from only the relevant ones → grade groundedness (is the answer actually supported?) → grade usefulness (does it actually address the question?) → return, regenerate once, or fall back honestly
• The genuinely new piece here: groundedness grading uses a local NLI (Natural Language Inference) model instead of another LLM call — meaningfully cheaper (~200-500ms locally vs. a full LLM round-trip) and, per recent benchmarking, comparably accurate for this specific kind of check
• A real, honest finding from building this: across several deliberate attempts to elicit a hallucination (incomplete context, no explicit refusal instruction, topically-adjacent-but-wrong context), Claude Sonnet 5 consistently declined rather than guessed — a genuinely useful data point about when this whole apparatus earns its keep with a frontier model, versus older/smaller/open models more prone to filling gaps
• Realistic cost: 4-6+ LLM calls per query, more if it loops — use a small/cheap model for the grading calls and reserve the expensive one for generation

Corrective RAG (CRAG)

• One batched evaluator call scores every retrieved chunk 1-5 for relevance, and the maximum score across the batch decides a 3-way verdict: Correct, Incorrect, or Ambiguous
• Each verdict triggers a genuinely different action, not just a different label: Correct refines the retrieved text down to the sentences that actually hold up; Incorrect discards it entirely and falls back to a real web search; Ambiguous does both and combines them
• Verified against real data, all three verdicts, including a live Tavily web search that returned actual current stock price data the sample paper obviously doesn't contain — and the model correctly cited it as an external source
• Realistic cost: 2-4 small/cheap-model calls plus, on the Incorrect/Ambiguous paths, one real network call to a search API — and exactly one expensive generation call, regardless of which branch runs

Agentic RAG

• The core mechanism is a ReAct loop (Reason, Act, Observe, repeat) — but built here with the model's actual tool-calling API, not prompted text parsed with regex or tags the way every earlier recipe's routing worked
• Four tools, each a real technique already verified elsewhere on this site: plain vector search, graph local search, graph global search, and a real web search fallback — the agent picks among them itself
• Verified correct on all 4 distinct question types, unprompted: a simple factual question, a broad synthesis question, a specific relational question, and a question entirely outside the corpus — no routing logic was written for any of it; the tool descriptions alone were enough
• This is genuinely more expensive and less predictable than everything before it — the number of tool calls varies per question, and a misinterpreted result can send the loop in an unproductive direction for several steps, which is why every real implementation needs a hard iteration cap and an honest "ran out of budget" exit
• Not free lunch, and not for everything — high-volume, well-understood query patterns are cheaper and more predictable with a fixed or modular pipeline; agentic control flow earns its cost specifically when the right retrieval strategy genuinely can't be known in advance

Graph RAG

• Building the graph is real work, measured honestly: on a 92-chunk sample paper, extraction alone took 92 LLM calls (one per chunk) and produced 605 entities and 819 relationships — that's before community detection or summarization even start
• Two query modes answer genuinely different question types: local search (entity-link the query, traverse the graph, generate from what's connected) for specific relational questions; global search (map-reduce over community summaries) for broad synthesis questions no single top-k retrieval can cover
• Both verified as real wins over naive retrieval, not asserted: local search found a connection naive retrieval missed at k=1 and k=3 (only surfacing at k=5); global search's answer to a broad architecture question covered components naive top-5 completely missed
• A real, honest limitation, caught and reported: entity deduplication by embedding similarity failed to merge "Encoder" and "the Encoder stack" — their actual similarity (0.833) fell just under the 0.88 threshold used
• Global search costs one LLM call per community, every single query — genuinely expensive; a newer approach (LazyGraphRAG) exists specifically to avoid this recurring cost

Specialized & Alternative Retrieval

Vision-RAG

• No text extraction anywhere: each PDF page is rendered as an image, embedded directly, and — when retrieved — read by the LLM as an image, not OCR'd text
• Late interaction (MaxSim), not a single vector: a page is embedded as a grid of patch vectors, not pooled into one summary vector, so a page can win retrieval on the strength of one region — a single table cell — even if the rest of the page is unrelated to the query
• Not a universal upgrade over naive text RAG: measured Recall@3 on 8 plain factual questions, naive chunk-based text RAG actually scores higher (1.000 vs. 0.875) — page-level retrieval is a coarser unit than a precise chunk, and on a mostly plain-text document, chunking's precision has room to win
• Where it genuinely wins: asked a question whose answer is a specific table cell, naive text RAG retrieves a passage that only mentions the table without containing its data, and the LLM correctly declines to guess; Vision RAG retrieves the table's page image and reads the correct value straight off it
• The smallest usable model in the ColPali family still needs a real fix to load: its packaged checkpoint config points at a training-time-only local path; the practical fix is a documented, reusable one — not a workaround to avoid mentioning

Multimodal RAG

• Two architectural approaches exist: a unified embedding space (a CLIP-family model puts text and images in the same vector space, so one similarity search covers both) or separate per-modality indexes combined by fusion
• Newer isn't automatically better — measure it: 3 candidate embedding models tested on the same corpus and questions. SigLIP (often cited as CLIP's successor) scored worse than base CLIP on both text and image retrieval. jina-clip-v2 doubled base CLIP's text Recall@3 (0.375 → 0.75) at the same image accuracy, and is the model used throughout this page
• The unified approach has a real cost, even with the best model tested: routing text through jina-clip-v2's text encoder instead of a dedicated text embedding model costs 25 points of Recall@3 on 8 factual questions, shown in full below
• Separate indexes + RRF fusion is the better default, verified two ways: it correctly retrieves both a diagram and its explanatory text for an image-appropriate question, and correctly returns only text for a plain factual question
• RRF is rank-based, not relevance-based: with a small image pool, the top-ranked image can win fusion on pure noise unless gated by a minimum-similarity threshold; and RRF produces an exact tie at rank 0 between modalities on every query — a structural property of rank fusion that no tiebreaker resolves

Video-RAG

• Two signals, one timestamp axis: a speech-to-text transcript (Whisper) and frames sampled at a fixed interval, both indexed by time, so a query can retrieve the relevant moment's transcript AND see what was on screen at that moment
• Measured directly: transcript-only retrieval scores 3/5 on a real eval set built from this page's own sample video — both misses have a distinct, checked cause, not a vague "retrieval sometimes fails"
• One failure mirrors ordinary chunk-boundary problems, just on the time axis: Whisper's own segmentation split a cause from its explanation across two adjacent segments — retrieval found the segment ending in "...begins to warm up because," but the actual reason was in the next one
• The other is embedding confusion between near-duplicates: two adjacent segments about similar-sounding percentages (90% vs. 100% of two different UV bands) crowded each other out of the top-3
• There's a real, verified case transcript-only retrieval cannot solve at all: a whiteboard axis label that's shown but never spoken — the transcript-only path correctly declines rather than guessing, and only passing the actual frame image gets the right answer
• This recipe deliberately doesn't index frames by visual similarity — frame grounding here is timestamp-proximity to the retrieved transcript segment, not a separate CLIP-based visual search (that mechanism is already covered on the Multimodal RAG and Vision RAG pages)

CAG (Cache-Augmented Generation)

• No retrieval at all: the entire document goes into the system prompt on every call — there's no embedding step, no vector search, no chunk selection anywhere in this pipeline
• "Fresh" (plain long-context stuffing) vs. "cached" (CAG's actual claim) are different things: sending the whole document every call is not CAG by itself — what makes it CAG is reusing the same processed prefix across queries instead of reprocessing it every time
• Cost reduction is dramatic and consistent, measured directly: cache reads are billed at 0.1x the base input price vs. 1.25x to write the cache and 1.0x with no caching at all — reprocessing a ~15K-token document on every call costs roughly 6-7x what reading it from cache costs, measured at 84-86% total cost reduction across two full runs
• Latency reduction was NOT reliably observed at this document size: the measured delta swung from +13% (cached actually slower) to -5% (cached faster) across two runs — noise-level, not a dependable win, even though Anthropic's own published figures claim up to 85% latency reduction at larger scale
• Caching can't change accuracy, and it didn't: the model reads byte-identical content either way, so any answer differences between runs are ordinary LLM phrasing variance, not a caching effect
• It doesn't scale by using an even bigger context window — a documented hybrid pattern ("RAG over CAG") exists for multi-document knowledge bases, with its own real trade-offs, covered below

PageIndex

• No embeddings, no vector database: a document's real structure becomes a tree; an LLM reasons over section titles and summaries to decide where to look, then pulls the actual text from just those sections
• A different sample document than every other page on this site, deliberately: the usual paper's shallow structure wouldn't exercise this technique's actual advantage, so this page uses a genuinely deep, 4-level-structured regulatory PDF instead
• Measured directly: PageIndex 5/5 correct with accurate page citations, naive text RAG 3/5, on a 5-question eval set built specifically for this document
• The two naive-RAG misses have a specific, checked cause: embedding similarity retrieved generically similar-sounding prose elsewhere in the document instead of the exact numbered subsection or exact appendix text — precisely the failure mode this technique targets
• The real cost: each PageIndex query here took 6-12 seconds (an actual reasoning loop over the tree), against naive RAG's well under a tenth of a second (one cosine-similarity lookup) — a real latency trade-off for the accuracy gain, not a free win

Retrieval Engineering

Chunking Strategies

• Default: recursive chunking, 256-512 tokens, ~10-20% overlap — cheap, structure-respecting, wins on end-to-end accuracy in 2026 benchmarks despite lower raw recall than semantic
• Chunking config rivals embedding-model choice in impact on retrieval quality (Vectara, NAACL 2025) — don't skip tuning this
• Semantic chunking gets higher recall (91.9% vs. recursive's 85-89%) but lower end-to-end accuracy (54% vs. 69%) — fragments can get topically pure but too small to actually answer from
• Small-to-big / parent-document is the broadly useful upgrade once precise retrieval and sufficient generation context are in tension — which is most real systems
• Late chunking flips the order entirely: embed the whole document first (full cross-attention), split the token embeddings after — chunks keep context from the rest of the document
• Proposition chunking goes finer than a sentence: one atomic, self-contained fact per unit (Dense X Retrieval, Chen et al. 2023)
• Agentic chunking costs an LLM API call per document; proposition chunking costs one local model inference call per chunk (free, no API) using the paper authors' released model — both still too slow for bulk indexing of a huge corpus
• Adaptive chunking doesn't pick a technique for you — it runs several, scores them, and picks per document, which matters once your corpus is genuinely mixed-format

Embedding Model Selection

• Hard rule: same embedding model for indexing and querying, always — mismatched models produce silently wrong rankings, not a crash (measured: 0.88 → 0.75 Recall@3 in a same-dimension mismatch)
• Matryoshka Representation Learning (MRL): one model, truncate the output vector to any smaller size with truncate_dim — no retraining, and quality often survives surprisingly well (OpenAI text-embedding-3, Gemini Embedding, Qwen3-Embedding, nomic-embed-text-v1.5 all support this)
• Quantization (int8/binary) is a separate, stackable lever from MRL — same dimension, smaller number format; binary is 32x smaller than float32 but can cost real recall (measured: 0.88 → 0.38 on a small eval set)
• Instruction-tuned embeddings expect a task-specific prefix at inference time — get the specific model's convention right (some prefix only the query, some prefix both sides differently) or silently lose quality
• Multi-vector / late-interaction (ColBERT) keeps one embedding per token, not per chunk, scored via MaxSim — a fundamentally different retrieval paradigm from every single-vector model
• Three retrieval paradigms, not one: BM25 (word counts, no model), dense embeddings (learned meaning, one vector per chunk), and SPLADE (learned meaning, shaped like BM25) each fail where the others succeed — worked comparison on a real 6-document example shows BM25 scoring the actual best answer a flat 0.000 while dense scores it highest
• BGE-M3 produces dense, sparse, and multi-vector representations from one model and one encode() call
• On MTEB, check the retrieval sub-score specifically, not overall rank — a model can top the leaderboard on classification while being mediocre at retrieval
• Fine-tune only after cheaper fixes are exhausted — hybrid search, reranking, and a better off-the-shelf model usually close more of the gap than fine-tuning does, for far less effort

Retrieval Methods

• MMR (Maximal Marginal Relevance) diversifies top-k, trading some pure relevance for less redundancy — a real, tested langchain_core utility, not something to hand-roll
• Self-query retrieval has an LLM split a query into a semantic part + structured metadata filters — but only works if the LLM is told the valid values for each field, not just the field names (measured: silently dropped a filter until the schema was explicit)
• Hypothetical Questions generates plausible questions per chunk at index time, so query-time comparison is question-to-question, not question-to-answer — the mirror image of HyDE, which instead generates a hypothetical answer at query time
• Hierarchical index retrieval narrows document → section → chunk in stages, instead of searching every chunk flat
• RAPTOR clusters chunks and summarizes each cluster, so broad synthesis questions can match a summary node instead of failing to match any single leaf chunk
• Adaptive retrieval (FLARE-style) only retrieves when the model's own self-reported confidence is low — measured: a too-permissive threshold let a wrong FLOPS figure ship ungrounded; a stricter one caught it, though even then, single-vector retrieval alone didn't find the actual answer

Vector Databases

• The core problem is approximate, not exact, nearest-neighbor search, and there are exactly three algorithm families worth knowing in depth: HNSW (graph-based, the default), IVF (cluster-based, needs training), and PQ (compresses the vectors themselves, usually paired with IVF)
• Every tunable parameter trades off recall against speed, memory, or build time — M/ef_construction/ef_search for HNSW, nlist/nprobe for IVF, m/nbits for PQ — this page explains what each one literally does, with real current defaults from each database's own source or docs
• Six real databases, same corpus, same metadata, both HNSW and IVF shown as runnable code — FAISS, ChromaDB, Qdrant, Milvus Lite, pgvector, and Weaviate, all indexed and queried against the same chunked PDF with each chunk's real page number as metadata
• Choosing a vector database is a system design question, not a feature-comparison exercise — the standard senior-level answer starts with "do you need one at all," reasons through a recall/latency/memory triangle (plus quantization as a fourth lever), and only escalates to a dedicated store once a concrete constraint demands it

Evaluation & Production

Response Quality & Safety

• Aggregate faithfulness scores hide per-claim hallucination — a response with 5 claims and 1 fabricated detail can still average ~0.8 on a faithfulness metric; the fix is claim-level decomposition (what RAGAS's Faithfulness metric already does), not a single answer-level verdict
• Anthropic's Citations API grounds attribution, not truth — it guarantees a claim came from an exact source passage, not that the claim is correct; a 2025 OpenAI/Georgia Tech paper argues hallucination persists because training-time benchmarks reward confident guessing over calibrated "I don't know" answers
• RAG's prompt-injection surface is the retrieved content itself, not just the user's message — EchoLeak (CVE-2025-32711) was a real, zero-click production exploit where a hidden instruction in an unopened email got pulled into Microsoft 365 Copilot's context and exfiltrated data
• Retrieval-time access control must be a hard metadata filter at the database layer, not a prompt instruction — the LLM cannot be relied on as a security boundary for content it was never supposed to see in the first place

Evaluation Metrics

• Retrieval metrics need ground truth — Precision@k, Recall@k, MRR, NDCG, and Hit Rate all require knowing which chunks are actually relevant to a query; this page builds that ground truth from a real substring match, not hand-picked page numbers
• Measured on a real system: dense retrieval at k=3 scored Precision@3 = 0.58, Recall@3 = 0.62, MRR = 0.81, NDCG@3 = 0.74, Hit Rate@3 = 0.88 across 8 questions — one question (attention head count) missed its relevant chunk entirely at k=3
• Classic generation metrics (BLEU, ROUGE-L, METEOR) measure word overlap, not correctness — this page's own measured run shows a factually wrong answer (41.0 instead of the correct 28.4 BLEU score) scoring BLEU=32.9, ROUGE-L=0.68, METEOR=0.76, and semantic similarity=0.89, because the wrong number sits inside an otherwise well-phrased, structurally similar sentence
• This page covers retrieval metrics and classic/embedding generation metrics; the LLM-judged groundedness diagnostic table (Faithfulness, Context Recall, etc.) is a separate, more involved measurement already covered with real code in RAGAS Deep Dive

RAGAS Deep Dive

• Four metrics, four different pipeline stages: Faithfulness and Answer Relevancy are generation-side, Context Precision is a ranking signal, Context Recall is a retrieval/indexing signal — the pattern across all four is meant to tell you exactly where to intervene, not just that quality dropped somewhere
• Needs a few version pins and one non-default wrapper flag — langchain-community==0.3.31, a LangchainEmbeddingsWrapper around a plain HuggingFace embeddings object instead of ragas's own, and bypass_temperature=True on the judge LLM wrapper, since current Claude models don't accept a temperature parameter
• Measured on a real naive RAG pipeline: aggregate scores of 0.91 faithfulness, 0.86 answer relevancy, 0.63 context precision, 0.88 context recall across the standard 8-question eval set
• The most useful result was the anomaly, not the average: one question got a factually correct answer, but scored near-zero on faithfulness and context recall — the model answered from its own training data after retrieval missed the actual table row, exactly the failure mode a plain "is the final answer right" check would miss entirely, and the pattern reproduced across separate runs
• Four more metrics are covered beyond the core diagnostic table — AnswerCorrectness, SemanticSimilarity, FactualCorrectness, and NoiseSensitivity from ragas's modern collections API, plus automatic synthetic eval-set generation via TestsetGenerator

Production Considerations

• Retries, fallbacks, and circuit breakers are three different tools, not one — retries handle transient failures, fallbacks route to a secondary provider, circuit breakers proactively stop sending requests to something that's already failing; this page's own circuit-breaker demo shows the real latency saved by not retrying a doomed call
• Semantic caching and prompt caching solve different problems — semantic caching skips the whole pipeline for a near-duplicate query (this page measured a real 33% hit rate, with a genuine miss right at the similarity threshold boundary); prompt caching (Anthropic's, mechanically confirmed) caches a shared prefix within calls that still each generate fresh output
• Standard uptime monitoring cannot see RAG's actual failure mode — a system serving wrong answers stays fast and returns 200 OK on every request, so retrieval-quality drift needs its own monitoring, not a byproduct of latency/error-rate dashboards
• Scalability at real production numbers is a GPU-throughput and region-placement problem for embedding, and a right-sized-model problem for every auxiliary LLM call — with real published numbers for both, not estimates

78 questions from every page on this site, one combined pass instead of opening each page separately. Every question shows which page it's from — go re-read that page for anything you get wrong.

184 flashcards from every page with a deck so far — click a card to flip it, shuffle for random order.