Skip to content

RAGAS Deep Dive

An open-source evaluation framework, not a single metric — a Python library computing a specific set of RAG-tailored metrics, most using an LLM as the underlying judge. RAGAS is a structured, purpose-built application of the LLM-as-judge pattern, specifically for RAG pipelines. This page runs it against a real pipeline's real output, not a curated example, to see whether its headline pitch — that the four core metrics together tell you exactly where a RAG system is failing — actually holds up.

Scope: ragas ships two parallel evaluation APIs, this page uses both

Current ragas (0.4.x) has a legacy evaluate() + EvaluationDataset + named-metric-class API — what every existing tutorial shows — and a newer per-sample ragas.metrics.collections + .ascore() API the current docs actually lead with. The legacy path still works (every import just fires a DeprecationWarning), and it's the only one that produces the four-metrics-together diagnostic table this page opens with — the modern collections classes can't be passed to evaluate() at all, since they don't share a base class with what evaluate() expects. Further down, this page also uses the modern collections API for four additional metrics and for synthetic eval-set generation, since neither is available through the legacy path.

Hands-on

The full pipeline below is runnable: RAGAS Deep Dive → in the code repo. Needs an Anthropic API key, used both for the naive RAG pipeline's own generation and as RAGAS's judge LLM.

TL;DR — quick revision
  • Four metrics, four different pipeline stages: Faithfulness and Answer Relevancy are generation-side, Context Precision is a ranking signal, Context Recall is a retrieval/indexing signal — the pattern across all four is meant to tell you exactly where to intervene, not just that quality dropped somewhere
  • Needs a few version pins and one non-default wrapper flag — langchain-community==0.3.31, a LangchainEmbeddingsWrapper around a plain HuggingFace embeddings object instead of ragas's own, and bypass_temperature=True on the judge LLM wrapper, since current Claude models don't accept a temperature parameter
  • Measured on a real naive RAG pipeline: aggregate scores of 0.91 faithfulness, 0.86 answer relevancy, 0.63 context precision, 0.88 context recall across the standard 8-question eval set
  • The most useful result was the anomaly, not the average: one question got a factually correct answer, but scored near-zero on faithfulness and context recall — the model answered from its own training data after retrieval missed the actual table row, exactly the failure mode a plain "is the final answer right" check would miss entirely, and the pattern reproduced across separate runs
  • Four more metrics are covered beyond the core diagnostic table — AnswerCorrectness, SemanticSimilarity, FactualCorrectness, and NoiseSensitivity from ragas's modern collections API, plus automatic synthetic eval-set generation via TestsetGenerator

The four metrics — precise mechanics

Faithfulness. Does the generated answer only contain claims actually supported by retrieved context? RAGAS breaks the answer into individual atomic claims (via LLM decomposition), then checks each claim against retrieved context individually. Score = supported claims ÷ total claims. Decomposing into claims rather than judging the whole answer at once means a 5-correct-1-unsupported answer gets a graded score, not one binary judgment that loses which specific claim is the problem.

Answer Relevancy. Does the generated answer actually address the question asked — independent of whether it's factually correct? Computed by generating several hypothetical questions the answer would be a good response to, then comparing those (by embedding cosine similarity) against the original question. This is deliberately the mirror image of HyDE: comparing question-to-question, not question-to-answer, avoids the same phrasing-asymmetry problem HyDE exists to fix on the retrieval side.

Context Precision. Of the chunks actually retrieved, are the relevant ones ranked near the top rather than buried lower? An LLM judges each retrieved chunk's relevance individually, and the metric rewards relevant chunks appearing early in the ranked list — mathematically similar in spirit to how NDCG rewards early-ranked relevance in general information retrieval. A low score with genuinely-relevant chunks somewhere in the results points at reranking, not retrieval recall.

Context Recall. Did retrieval capture everything actually needed to answer correctly? This is the one metric that needs a ground-truth reference answer: the reference gets decomposed into claims the same way Faithfulness decomposes the generated answer, and each reference claim is checked against the retrieved context (not the generated answer). A low score means retrieval genuinely failed to find necessary information — a retrieval/indexing problem, not a generation problem, since the information was never in the retrieved set to begin with.

Running it: naive RAG pipeline + RAGAS scoring

def run_naive_rag(question: str, chunks: list, reference: str) -> dict:
    """A real naive RAG call: retrieve top-3 chunks, generate an answer,
    package it into the shape RAGAS needs -- question, answer, the
    retrieved contexts (not just the final answer), and a reference
    (ground-truth) answer for the metrics that need one."""
    import anthropic
    from sentence_transformers import SentenceTransformer, util

    model = SentenceTransformer("all-MiniLM-L6-v2")
    chunk_embeddings = model.encode(chunks, convert_to_numpy=True, show_progress_bar=False)
    query_embedding = model.encode(question, convert_to_numpy=True)
    sims = util.cos_sim(query_embedding, chunk_embeddings)[0]
    top_k = sims.topk(min(3, len(chunks)))
    retrieved_contexts = [chunks[i] for i in top_k.indices.tolist()]

    client = anthropic.Anthropic()
    context_block = "\n---\n".join(retrieved_contexts)
    response = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=1024,
        messages=[{"role": "user", "content": f"Answer the question using the context below.\n\nContext:\n{context_block}\n\nQuestion: {question}"}],
    )
    answer = response.content[0].text

    return {"question": question, "answer": answer, "retrieved_contexts": retrieved_contexts, "reference": reference}

def run_ragas_eval(samples: list) -> object:
    """Scores real pipeline output with RAGAS's four core metrics.
    `samples` is a list of dicts shaped like `run_naive_rag`'s return
    value.

    Setup requirements for this ragas release: embeddings go through
    `LangchainEmbeddingsWrapper` around a real `langchain-huggingface`
    embeddings object, the same pattern used for the judge LLM via
    `LangchainLLMWrapper` -- ragas's own `HuggingfaceEmbeddings` wrapper
    fails pydantic validation on direct instantiation. The judge LLM
    wrapper also needs `bypass_temperature=True`, since current Claude
    models (Sonnet 5 and later) no longer accept a `temperature`
    parameter, which `LangchainLLMWrapper` otherwise sets on the
    underlying model before every call by default.
    """
    from langchain_anthropic import ChatAnthropic
    from langchain_huggingface import HuggingFaceEmbeddings
    from ragas import EvaluationDataset, RunConfig, SingleTurnSample, evaluate
    from ragas.embeddings import LangchainEmbeddingsWrapper
    from ragas.llms import LangchainLLMWrapper
    from ragas.metrics import (
        Faithfulness,
        LLMContextPrecisionWithReference,
        LLMContextRecall,
        ResponseRelevancy,
    )

    evaluator_llm = LangchainLLMWrapper(ChatAnthropic(model="claude-sonnet-5"), bypass_temperature=True)
    evaluator_embeddings = LangchainEmbeddingsWrapper(HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2"))

    dataset = EvaluationDataset(samples=[
        SingleTurnSample(
            user_input=s["question"],
            response=s["answer"],
            retrieved_contexts=s["retrieved_contexts"],
            reference=s["reference"],
        )
        for s in samples
    ])

    metrics = [
        Faithfulness(llm=evaluator_llm),
        ResponseRelevancy(llm=evaluator_llm, embeddings=evaluator_embeddings),
        LLMContextPrecisionWithReference(llm=evaluator_llm),
        LLMContextRecall(llm=evaluator_llm),
    ]

    # Anthropic's per-minute rate limits are tighter than ragas's 16-worker
    # default assumes.
    return evaluate(dataset=dataset, metrics=metrics, run_config=RunConfig(max_workers=4, timeout=180))

Setup requirements for this ragas release

langchain-community==0.3.31 needs pinning — ragas's own code imports langchain_community.chat_models.vertexai, a module later langchain-community releases removed (moved to a separate langchain-google-vertexai package), and ragas carries no upper pin on it.

Embeddings go through LangchainEmbeddingsWrapper around a plain langchain-huggingface embeddings object, the same pattern used for the judge LLM via LangchainLLMWrapper — ragas's own HuggingfaceEmbeddings class fails pydantic validation on direct instantiation in this release.

The judge LLM wrapper needs bypass_temperature=True. LangchainLLMWrapper sets a temperature value on the underlying model before every call by default, and current Claude models no longer accept that parameter (they use adaptive thinking instead of sampling controls) — bypass_temperature is the wrapper's own documented flag for LLMs like this.

Measured: does the diagnostic table hold up on a real system?

Aggregate scores across the standard 8-question eval set used throughout this site:

faithfulness:                          0.9062
answer_relevancy:                      0.8596
llm_context_precision_with_reference:  0.6250
context_recall:                        0.8750

Putting the four metrics together — diagnosing a RAG system

The four metrics map to different pipeline stages, so their pattern tells you where to intervene:

Faithfulness Answer Relevancy Context Precision Context Recall Diagnosis
Low — — High Generation is hallucinating or ignoring good context — a generation/prompting problem
— Low — — Answer doesn't address the question asked — a generation/prompting problem (wrong sub-question, drifting)
High High Low High Right info retrieved but ranked poorly — a reranking problem
High High — Low Necessary info not retrieved at all — a retrieval/indexing problem
High High High High System working well across the whole pipeline

A factually correct answer with near-zero grounding

Asked "What BLEU score did they get on English-to-German translation?", the naive RAG pipeline answered "the Transformer model achieved a BLEU score of 28.4" — factually correct, 28.4 is the real number. But that question has repeatedly scored near-zero on faithfulness, context precision, and context recall across separate runs of this pipeline (0.167/0.0/0.0 in one run, 0.25/0.0/0.0 in another). A context_recall of 0.0 means the retrieved context didn't actually support the reference claim; in the first run, checking the actual top-3 retrieved chunks confirmed exactly why — one chunk was the results table cut off right before the Transformer's own rows appear, and the other two both discussed the English-to-French numbers (41.0, 41.8) from a different part of the paper. None of the three retrieved chunks contained "28.4" at all. Retrieval genuinely missed the right table row, both times. The model answered correctly anyway — from its own training data on this well-known paper, not from what was actually retrieved.

This is the diagnostic table's "High-looking answer, but check faithfulness and recall before trusting it" case in the wild, and it isn't a one-off fluke: a factually right answer with near-zero grounding in what was actually retrieved, which a simple "is the final answer correct" check would have missed completely both times. It's also a real, measured argument for why Faithfulness and Context Recall exist as separate metrics from plain answer accuracy — accuracy alone would have marked this a pass on both runs.

Beyond the core four: additional metrics

ragas.metrics.collections — the modern per-sample API the scope note above describes — ships a much larger catalog than the four-metric diagnostic table: classic NLP metrics (BleuScore, RougeScore), semantic and factual checks (SemanticSimilarity, FactualCorrectness, AnswerCorrectness), NoiseSensitivity, ContextEntityRecall, rubric-based custom metrics, and agentic-workflow metrics (ToolCallAccuracy, TopicAdherence, goal-accuracy variants) for systems this page doesn't cover. Four of these — the ones with the most direct overlap with the core four — are measured here against the same 8-question eval set:

  • AnswerCorrectness — a weighted blend of factual correctness and semantic similarity against the reference answer.
  • SemanticSimilarity — embedding cosine similarity between generated answer and reference, on its own.
  • FactualCorrectness — decomposes both answer and reference into claims and checks them against each other via NLI, independent of retrieved context.
  • NoiseSensitivity — how much an answer's correctness degrades when the retrieved context contains irrelevant ("noisy") chunks alongside relevant ones.
async def run_extended_metrics(sample: dict, judge_model: str = "claude-sonnet-5") -> dict:
    """Scores one sample with four metrics from ragas's modern
    `ragas.metrics.collections` API -- AnswerCorrectness (a weighted
    blend of factual correctness and semantic similarity),
    SemanticSimilarity on its own, FactualCorrectness, and
    NoiseSensitivity. `sample` needs question/answer/reference/
    retrieved_contexts keys, matching `run_naive_rag`'s return shape.

    This API builds its LLM via `llm_factory` + the `instructor`
    library rather than LangChain, so `LangchainLLMWrapper`'s
    `bypass_temperature` flag doesn't apply here -- current Claude
    models reject both `temperature` and `top_p`, and this API has no
    equivalent constructor flag, so both are removed directly from the
    constructed LLM's own `model_args` dict. Needs `instructor>=1.17`;
    older versions fail to parse a response when the model returns a
    ThinkingBlock before its text content.
    """
    from anthropic import AsyncAnthropic
    from ragas.embeddings import HuggingFaceEmbeddings
    from ragas.llms import llm_factory
    from ragas.metrics.collections import AnswerCorrectness, FactualCorrectness, NoiseSensitivity, SemanticSimilarity

    def make_judge_llm():
        llm = llm_factory(judge_model, provider="anthropic", client=AsyncAnthropic())
        del llm.model_args["temperature"]
        del llm.model_args["top_p"]
        return llm

    embeddings = HuggingFaceEmbeddings(model="sentence-transformers/all-MiniLM-L6-v2", use_api=False)

    correctness = AnswerCorrectness(llm=make_judge_llm(), embeddings=embeddings)
    similarity = SemanticSimilarity(embeddings=embeddings)
    factual = FactualCorrectness(llm=make_judge_llm())
    noise = NoiseSensitivity(llm=make_judge_llm())

    answer_correctness = await correctness.ascore(user_input=sample["question"], response=sample["answer"], reference=sample["reference"])
    semantic_similarity = await similarity.ascore(reference=sample["reference"], response=sample["answer"])
    factual_correctness = await factual.ascore(response=sample["answer"], reference=sample["reference"])
    noise_sensitivity = await noise.ascore(
        user_input=sample["question"], response=sample["answer"], reference=sample["reference"], retrieved_contexts=sample["retrieved_contexts"],
    )
    return {
        "answer_correctness": answer_correctness.value,
        "semantic_similarity": semantic_similarity.value,
        "factual_correctness": factual_correctness.value,
        "noise_sensitivity": noise_sensitivity.value,
    }

Measured aggregate scores, same 8-question eval set:

answer_correctness:   0.7238
semantic_similarity:  0.7899
factual_correctness:  0.4600
noise_sensitivity:    0.5089

FactualCorrectness is measurably stricter than semantic similarity

On the "What GPUs was the model trained on?" question, the naive RAG pipeline answered "The model was trained on 8 NVIDIA P100 GPUs" against a reference of "The models were trained on NVIDIA P100 GPUs" — the same fact, phrased slightly differently. SemanticSimilarity scored this pair 0.877, but FactualCorrectness scored it 0.0. FactualCorrectness's claim-decomposition-plus-NLI check is stricter about scope and phrasing than embedding similarity, penalizing differences (singular vs. plural framing, an added specific number) a human reader would likely still call factually equivalent. Worth knowing before treating a low FactualCorrectness score alone as proof of an actual factual error, rather than a stricter check flagging a phrasing mismatch.

Setup for the modern collections API

This API builds its LLM via llm_factory + the instructor library rather than LangChain, so LangchainLLMWrapper's bypass_temperature flag doesn't apply. Current Claude models reject both temperature and top_p, and this API has no equivalent constructor flag — both need removing directly from the constructed LLM's own model_args dict, as make_modern_judge_llm() above does. Separately, instructor>=1.17 is required against Anthropic models specifically — older versions fail to parse a response when the model returns a ThinkingBlock before its text content.

Generating an eval set automatically

Every measurement on this page needed a hand-written eval set with real reference answers. ragas.testset.TestsetGenerator builds one automatically from source documents instead — useful when a reference eval set doesn't exist yet.

def generate_testset(document_text: str, testset_size: int = 6, judge_model: str = "claude-sonnet-5") -> object:
    """Generates a synthetic eval set (question + reference answer
    pairs) directly from a source document, for when there's no
    hand-written eval set yet. Uses the legacy LangchainLLMWrapper/
    LangchainEmbeddingsWrapper, not the modern collections API, since
    TestsetGenerator's constructor expects those wrapper types
    specifically. Needs the optional `rapidfuzz` package for its
    internal relationship-builder transform."""
    from langchain_anthropic import ChatAnthropic
    from langchain_core.documents import Document as LCDocument
    from langchain_huggingface import HuggingFaceEmbeddings
    from ragas.embeddings import LangchainEmbeddingsWrapper
    from ragas.llms import LangchainLLMWrapper
    from ragas.testset import TestsetGenerator

    docs = [LCDocument(page_content=document_text)]
    llm = LangchainLLMWrapper(ChatAnthropic(model=judge_model), bypass_temperature=True)
    embeddings = LangchainEmbeddingsWrapper(HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2"))
    generator = TestsetGenerator(llm=llm, embedding_model=embeddings)
    return generator.generate_with_langchain_docs(docs, testset_size=testset_size)

Run against the full sample paper, real generated questions include both clean, formal phrasing and a genuine typo-laden variant — TestsetGenerator deliberately varies persona and query style, not just topic:

"As I examine the foundational contributions behind the Transformer architecture, I am curious to
understand precisely what specific innovations Noam Shazeer proposed..."

"wat did noam shazer propse in teh transformr paper"

Both are real generated questions from the same run, about the same underlying content — a genuine argument for synthetic generation producing a more realistic eval set than a hand-written one, which rarely includes deliberately messy phrasing.

The multi-hop synthesizer needs more cross-document structure than one paper provides

Every run against this page's sample paper skips the multi-hop synthesizer entirely (No relationships match the provided condition. Cannot form clusters.), even against the full ~40K-character document — a single technical paper doesn't have enough thematic or entity overlap between different sections for the relationship-builder to find valid multi-hop clusters on. All generated questions come from the single-hop synthesizers instead. This isn't a bug; multi-hop generation needs a corpus with genuine cross-document relationships (e.g. multiple related papers or a multi-file knowledge base) to have anything to connect.

Practical considerations for using RAGAS in production

LLM choice for the judge matters and costs money. Most RAGAS metrics use an LLM internally for claim extraction and relevance judgments — cost scales with the number of claims and chunks evaluated, for every eval-set item. Typically run against a curated eval set of dozens to a few hundred examples, not on every live production query in real time.

Context Recall specifically requires ground-truth reference answers — a labeled eval set with correct answers, more work to build than the reference-free metrics (Faithfulness, Answer Relevancy, and Context Precision can all run without ground truth, using only the question, retrieved context, and generated answer).

RAGAS scores are still LLM judgments, not ground truth themselves — the same caveats as any LLM-as-judge approach (verbosity bias, judge-model quality ceiling) apply. Worth validating against a human-eval sample before trusting scores at scale, same as any other automated metric.

Use RAGAS for regression testing, not just one-time scoring. Run against a fixed eval set every time chunking, embedding model, reranker, or prompt changes — turns RAGAS into a CI-style regression test for RAG quality, catching a change that improves one metric while quietly regressing another.

Scenario Check

Sources & further reading