Self-RAG

Even with good retrieval, reranking, and compression, a generation LLM can still: generate an answer when it should have retrieved but didn't, use irrelevant retrieved content just because it's sitting in the prompt, or produce an answer that isn't actually supported by the retrieved context — hallucination despite having the right documents right there. Every earlier technique on this site improves what gets retrieved. Self-RAG is about making the model critique its own process and output instead of just consuming whatever it's handed.

Hands-on

The full loop below is runnable: Self-RAG → in the code repo. Relevance grading, generation, and usefulness grading all need an LLM key; the groundedness check is fully local (no key needed for that specific stage).

TL;DR — quick revision

• Two implementation paths exist, and they're easy to conflate: the original paper fine-tunes a model to emit special reflection tokens; almost everyone in production instead uses a prompted version — a pipeline of separate LLM calls doing the same grading jobs, no fine-tuning at all. Both are legitimately "Self-RAG."
• The prompted loop, in order: retrieve → grade each chunk's relevance → generate from only the relevant ones → grade groundedness (is the answer actually supported?) → grade usefulness (does it actually address the question?) → return, regenerate once, or fall back honestly
• The genuinely new piece here: groundedness grading uses a local NLI (Natural Language Inference) model instead of another LLM call — meaningfully cheaper (~200-500ms locally vs. a full LLM round-trip) and, per recent benchmarking, comparably accurate for this specific kind of check
• A real, honest finding from building this: across several deliberate attempts to elicit a hallucination (incomplete context, no explicit refusal instruction, topically-adjacent-but-wrong context), Claude Sonnet 5 consistently declined rather than guessed — a genuinely useful data point about when this whole apparatus earns its keep with a frontier model, versus older/smaller/open models more prone to filling gaps
• Realistic cost: 4-6+ LLM calls per query, more if it loops — use a small/cheap model for the grading calls and reserve the expensive one for generation

Two implementation paths — trained vs. prompted

Path 1 — trained (the original 2023 Self-RAG paper): the target LLM is fine-tuned to emit special reflection tokens as literal tokens in its own output — [Retrieve: Yes/No], [IsRel: Relevant/Irrelevant], [IsSup: Fully/Partially/No Support], [IsUse: 1-5] — interleaved with generated text. A separate "critic" model (e.g. GPT-4, prompted with a rubric) labels a large corpus first; the target model is then fine-tuned with standard next-token-prediction loss over sequences that include these tokens at the right points. Nothing exotic about the training objective — the special part is that the tokens exist at all.

Path 2 — prompted (what most real implementations use): no fine-tuning. A pipeline of separate LLM calls, each doing one grading job via structured output, standing in for what the trained tokens would decide. This is the dominant approach in practice — the standard LangGraph reference implementation, and most reproductions on models with no trained/reflection-token version available, use this pattern. It's legitimately Self-RAG, not a lesser substitute — and it's what this page's code builds.

The prompted loop, piece by piece

Naive RAG does four things blindly: it retrieves a fixed top-k with no check on whether any of it is actually useful, it hands the LLM everything it retrieved even if some of it is off-topic, it generates without confirming the answer is actually backed by what was retrieved, and it returns whatever comes out even if that answer doesn't really address the question. Self-RAG inserts an explicit check at each of those four blind spots. To make this concrete, one real question — "What optimizer did they use for training?" — is followed through all five stages below, with the actual intermediate output at each one.

1. Grade relevance — don't hand the LLM everything you retrieved

Bi-encoder retrieval returns the top-k by similarity, but "similar enough to rank in the top-k" and "actually useful for answering this question" aren't the same thing — a chunk can rank highly because it shares vocabulary or topic with the query without actually containing the answer. This step checks each retrieved chunk individually, before generation ever sees it, and throws out the ones that don't hold up.

import re

import llm

RELEVANCE_PROMPT = """Is the passage below relevant to answering the \
question? A passage is relevant if it contains information that would \
help answer the question, even partially.

Question: {question}

Passage:
{chunk}

Respond with ONLY yes or no, wrapped in <relevant></relevant> tags."""


def grade_relevance(question: str, chunk: str, provider: str | None = None) -> bool:
    response = llm.generate(RELEVANCE_PROMPT.format(question=question, chunk=chunk), provider=provider)
    match = re.search(r"<relevant>(.*?)</relevant>", response, re.S)
    return bool(match) and match.group(1).strip().lower() == "yes"

What actually happened, for the running example — naive retrieval pulled 5 chunks; relevance grading kept 2 and discarded 3:

chunk 0 (relevant=False): "warmup_steps = 4000. 5.4 Regularization We employ
  three types of regularization during training..."
chunk 1 (relevant=True):  "(3.5 days). 5.3 Optimizer We used the Adam optimizer
  [20] with β1 = 0.9, β2 = 0.98 and ϵ = 10−9..."
chunk 2 (relevant=True):  "target tokens. 5.2 Hardware and Schedule We trained
  our models on one machine with 8 NVIDIA P100 GPUs..."
chunk 3 (relevant=False): "Table 2: The Transformer achieves better BLEU
  scores than previous state-of-the-art models..."
chunk 4 (relevant=False): "were written at 10-minute intervals. For the big
  models, we averaged the last 20 checkpoints..."

Worth noting honestly: chunk 2 (GPU hardware, not the optimizer) got graded relevant too — a judgment call, not a wrong one, since it's from the same training-setup section and could plausibly help contextualize the answer. This step is a real LLM judgment, not a keyword filter, and it won't always draw the line exactly where you would.

2. Generate — but only from what survived grading

The retrieved-and-filtered chunks (chunk 1 and chunk 2 from above) become the only context the model sees for this question — chunks 0, 3, and 4 are gone, so nothing about regularization, BLEU scores, or checkpoint averaging can leak into or distract this answer.

import llm

GENERATE_PROMPT = """Answer the question using ONLY the context below. \
If the context doesn't contain enough to answer, say so explicitly \
rather than guessing.

Context:
{context}

Question: {question}"""


def generate_answer(question: str, chunks: list[str], provider: str | None = None) -> str:
    context = "\n---\n".join(chunks) if chunks else "(no relevant context found)"
    return llm.generate(GENERATE_PROMPT.format(context=context, question=question), provider=provider)

For the running example, this produces: "According to the context, they used the Adam optimizer with β1 = 0.9, β2 = 0.98 and ϵ = 10^-9." Note the prompt explicitly tells the model to say so rather than guess when the context is thin — the same discipline used in naive RAG's prompt. That instruction is what's doing the work in the fallback trace further down this page, where the model correctly declines instead of guessing.

3. Grade groundedness — is the answer actually backed by that context, or did the model add something?

This is the check for the specific failure naive RAG can't catch: an answer that sounds right and uses the right vocabulary, but says something the retrieved text doesn't actually support — a number changed, a claim added, a detail from the model's own general knowledge slipped in instead of what was in front of it. Most write-ups implement this as another LLM call asking "is this supported?" — this recipe uses a local NLI (Natural Language Inference) cross-encoder instead, treating the retrieved context as a premise and the generated answer as a hypothesis, and asking whether the premise entails it.

import numpy as np
from sentence_transformers import CrossEncoder

NLI_MODEL_NAME = "cross-encoder/nli-deberta-v3-base"


def grade_groundedness(answer: str, chunks: list[str]) -> dict:
    """premise = the relevant retrieved context (concatenated),
    hypothesis = the generated answer. "entailment" means the context
    actually supports the answer; "contradiction" or "neutral" both
    count as NOT grounded — either the context disagrees with the
    answer, or it simply doesn't say enough to support it.

    This uses a local NLI (Natural Language Inference) cross-encoder
    instead of an LLM-judge call for this specific check — locally,
    in ~200-500ms, vs. a full LLM round-trip — and is comparably
    accurate for this kind of QA-over-retrieved-context faithfulness
    check specifically (per recent benchmarking; it's a worse fit for
    open-ended generation with no clear source document).

    Known simplification: this checks the answer as ONE hypothesis
    against the whole context as one premise, rather than decomposing
    the answer into individual claims and checking each separately —
    enough to demonstrate the mechanism, but can miss a
    partially-ungrounded multi-claim answer where most of it is
    supported and only one clause isn't."""
    reranker = CrossEncoder(NLI_MODEL_NAME)

    if not chunks:
        return {"grounded": False, "label": "no_context"}

    context = " ".join(chunks)
    scores = reranker.predict([(context, answer)])[0]
    label = reranker.config.id2label[int(np.argmax(scores))]
    return {"grounded": label == "entailment", "label": label}

For the running example: context = chunks 1 and 2 above, answer = the Adam-optimizer sentence from step 2 → the model outputs entailment, so grounded=True. To see the check actually distinguishing something, here's a controlled test against the same context with a deliberately wrong claim swapped in:

Context: "We used the Adam optimizer with beta1=0.9, beta2=0.98."

Answer: "The model was trained using the Adam optimizer."
-> entailment (grounded=True)

Answer: "The model was trained using stochastic gradient descent (SGD)."
-> contradiction (grounded=False)

Only entailment counts as grounded — both contradiction (the context actively disagrees) and neutral (the context just doesn't say enough either way) mean the answer isn't backed by what was retrieved, and the pipeline treats them the same way: not good enough to return as-is.

4. Grade usefulness — grounded isn't the same as helpful

An answer can pass the groundedness check and still fail the user: "I cannot be completely certain about training details without more information" is not contradicted by anything, but it also doesn't answer the question — grounded, yet useless. This step catches that gap.

import re

import llm

USEFULNESS_PROMPT = """Does the answer below actually address the \
question — regardless of whether it's factually correct? An answer \
that's grounded but off-topic, or that just says "I don't know" when \
a real answer was possible, is NOT useful.

Question: {question}

Answer: {answer}

Respond with ONLY yes or no, wrapped in <useful></useful> tags."""


def grade_usefulness(question: str, answer: str, provider: str | None = None) -> bool:
    response = llm.generate(USEFULNESS_PROMPT.format(question=question, answer=answer), provider=provider)
    match = re.search(r"<useful>(.*?)</useful>", response, re.S)
    return bool(match) and match.group(1).strip().lower() == "yes"

For the running example, the Adam-optimizer answer from step 2 grades useful=True — it directly answers what was asked. The hedge sentence above, tested directly against this same grader, comes back False. Groundedness and usefulness are deliberately separate checks because they can fail independently: grounded-but-useless (the hedge), or — in principle — useful-but-ungrounded (a confident, on-topic answer that's simply wrong).

5. The full loop — return, retry, or fall back honestly

All four stages above are wired together here: if an answer passes both groundedness and usefulness, it's returned immediately (that's what happens for the running example — one pass, no retry needed). If either check fails, the pipeline regenerates once and re-checks. If it's still not grounded and useful after that, it stops trying and returns an explicit "I don't have enough reliably-supported information" message instead of a second silent failure — the same "flag rather than guess again forever" pattern real production Self-RAG systems use.

import numpy as np
from sentence_transformers import CrossEncoder, SentenceTransformer

import llm

MAX_REGENERATIONS = 1


def _naive_retrieve(question, chunks, embed_model, k):
    query_vec = embed_model.encode(question, convert_to_numpy=True)
    chunk_vecs = embed_model.encode(chunks, convert_to_numpy=True, show_progress_bar=False)
    sims = [float(np.dot(query_vec, v) / (np.linalg.norm(query_vec) * np.linalg.norm(v) + 1e-8)) for v in chunk_vecs]
    top_indices = np.argsort(sims)[::-1][:k]
    return [chunks[i] for i in top_indices]


def self_rag(
    question: str,
    chunks: list[str],
    embed_model: SentenceTransformer,
    provider: str | None = None,
    k: int = 5,
) -> dict:
    """Retrieve -> grade relevance -> generate -> grade groundedness ->
    grade usefulness -> return, regenerate once, or fall back to an
    honest "insufficient support" response instead of a second silent
    failure. See grade_relevance, generate_answer, grade_groundedness,
    and grade_usefulness above for each stage's own docs."""
    import re as _re

    reranker = CrossEncoder("cross-encoder/nli-deberta-v3-base")

    retrieved = _naive_retrieve(question, chunks, embed_model, k)

    def _relevant(q, c):
        resp = llm.generate(
            f"Is the passage below relevant to answering the question? A passage is relevant "
            f"if it contains information that would help answer the question, even partially.\n\n"
            f"Question: {q}\n\nPassage:\n{c}\n\n"
            f"Respond with ONLY yes or no, wrapped in <relevant></relevant> tags.",
            provider=provider,
        )
        m = _re.search(r"<relevant>(.*?)</relevant>", resp, _re.S)
        return bool(m) and m.group(1).strip().lower() == "yes"

    relevant = [c for c in retrieved if _relevant(question, c)]
    trace = {"retrieved": len(retrieved), "relevant": len(relevant), "attempts": []}

    for _ in range(MAX_REGENERATIONS + 1):
        context = "\n---\n".join(relevant) if relevant else "(no relevant context found)"
        answer = llm.generate(
            f"Answer the question using ONLY the context below. If the context doesn't "
            f"contain enough to answer, say so explicitly rather than guessing.\n\n"
            f"Context:\n{context}\n\nQuestion: {question}",
            provider=provider,
        )

        if not relevant:
            groundedness = {"grounded": False, "label": "no_context"}
        else:
            scores = reranker.predict([(" ".join(relevant), answer)])[0]
            label = reranker.config.id2label[int(np.argmax(scores))]
            groundedness = {"grounded": label == "entailment", "label": label}

        useful_resp = llm.generate(
            f'Does the answer below actually address the question — regardless of whether '
            f'it\'s factually correct? An answer that\'s grounded but off-topic, or that just '
            f'says "I don\'t know" when a real answer was possible, is NOT useful.\n\n'
            f"Question: {question}\n\nAnswer: {answer}\n\n"
            f"Respond with ONLY yes or no, wrapped in <useful></useful> tags.",
            provider=provider,
        )
        m = _re.search(r"<useful>(.*?)</useful>", useful_resp, _re.S)
        useful = bool(m) and m.group(1).strip().lower() == "yes"

        trace["attempts"].append({"answer": answer, "groundedness": groundedness, "useful": useful})

        if groundedness["grounded"] and useful:
            trace["final_answer"] = answer
            trace["status"] = "ok"
            return trace

    trace["final_answer"] = (
        "I don't have enough reliably-supported information in the retrieved context to answer this confidently."
    )
    trace["status"] = "fell_back"
    return trace

A real, unforced end-to-end trace — the fallback working as intended

Asked "What company is the author who proposed replacing RNNs with self-attention affiliated with?" — a fact already known, from the pre-retrieval page's IRCoT testing, to be genuinely unfindable by any query phrasing on this corpus:

Retrieved: 5, graded relevant: 0

--- attempt 1 ---
Answer: I don't have enough context to answer this question...
Groundedness: no_context (grounded=False)
Useful: False

--- attempt 2 ---
Answer: I cannot answer this question because no relevant context was provided...
Groundedness: no_context (grounded=False)
Useful: False

Status: fell_back
Final answer: I don't have enough reliably-supported information in the
retrieved context to answer this confidently.

Every stage did its job correctly on a genuine retrieval failure, not a manufactured one: relevance grading discarded all 5 candidates as irrelevant, generation honestly declined rather than guessing, both checks correctly flagged the result, the loop retried once, and the pipeline surfaced an honest final message instead of a confident wrong answer. This is the actual value proposition of the whole apparatus, working exactly as intended.

A real, honest finding: this model didn't want to hallucinate

Building the groundedness demo required deliberately trying to elicit an ungrounded answer — several constructions were tested: context that omits the requested fact entirely, prompts with no explicit "say so if missing" instruction, and topically-related-but-non-answering context. Claude Sonnet 5 declined every single time, across repeated trials, rather than filling in a plausible-sounding answer from its own pretraining knowledge (which, for a paper this famous, it almost certainly has memorized). That's a genuinely useful data point, not a failed experiment: it suggests the marginal value of a separate groundedness-checking stage is smaller with a frontier, safety-tuned model like this one than it would be with an older, smaller, or less-aligned model more prone to filling gaps confidently. The checks were still verified directly — see the controlled tests above — and remain valuable for less-aligned models, adversarial inputs, or domains where you can't risk even a rare failure.

Comparison to LLM-as-judge

LLM-as-judge is typically a separate model/call scoring output after the fact, outside the generation pipeline — used for offline evaluation, not something the system itself acts on. Self-RAG integrates the same kind of judgment into the pipeline itself, at generation time, so the system can act on it: discard irrelevant chunks, regenerate, or fall back — rather than just measuring quality after the answer has already gone out.

Use it for / skip it for

Use it for: high-stakes QA where a wrong or unsupported answer is costly (legal, medical, financial, compliance) and the extra latency/cost of multiple calls per query is acceptable. Skip it for: high-throughput, low-latency, simple-lookup use cases where naive RAG or a single reranking pass is already good enough — realistic cost here is 4-6+ LLM calls per query, more on a retry, and that's not free.

Scenario Check

Sources & further reading

• Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection — the original paper, trained reflection tokens
• LangGraph's Self-RAG reference implementation — the standard prompted pattern this recipe follows