Evaluation Metrics¶
RAGAS packages a specific, opinionated set of metrics behind an LLM judge. This page covers the metrics underneath and around that: standard information-retrieval metrics for the retrieval half of a RAG system, and the classic lexical-overlap metrics for the generation half — computed with real code against a real corpus and real generated answers, not described in the abstract.
Hands-on
The full pipeline below is runnable: Evaluation Metrics → in the code repo. No paid API needed — generation runs through a local Ollama model.
TL;DR — quick revision
- Retrieval metrics need ground truth — Precision@k, Recall@k, MRR, NDCG, and Hit Rate all require knowing which chunks are actually relevant to a query; this page builds that ground truth from a real substring match, not hand-picked page numbers
- Measured on a real system: dense retrieval at k=3 scored Precision@3 = 0.58, Recall@3 = 0.62, MRR = 0.81, NDCG@3 = 0.74, Hit Rate@3 = 0.88 across 8 questions — one question (attention head count) missed its relevant chunk entirely at k=3
- Classic generation metrics (BLEU, ROUGE-L, METEOR) measure word overlap, not correctness — this page's own measured run shows a factually wrong answer (41.0 instead of the correct 28.4 BLEU score) scoring BLEU=32.9, ROUGE-L=0.68, METEOR=0.76, and semantic similarity=0.89, because the wrong number sits inside an otherwise well-phrased, structurally similar sentence
- This page covers retrieval metrics and classic/embedding generation metrics; the LLM-judged groundedness diagnostic table (Faithfulness, Context Recall, etc.) is a separate, more involved measurement already covered with real code in RAGAS Deep Dive
Retrieval metrics¶
All five metrics below need the same input: for a given query, a ranked list of retrieved chunk IDs, and the real set of chunks that are actually relevant to it. Getting that ground truth right matters more than the metric formulas — a wrong relevance set produces confident, meaningless numbers.
Precision@k. Of the top-k retrieved chunks, what fraction are actually relevant? relevant_retrieved / k. Penalizes retrieving noise.
Recall@k. Of all relevant chunks that exist in the corpus for a query, what fraction did the top-k retrieval actually catch? relevant_retrieved / total_relevant. Penalizes missing relevant content entirely, independent of how much noise came with it.
MRR (Mean Reciprocal Rank). For each query, find the rank of the first relevant result — rank 3 gives a reciprocal rank of 1/3 — then average across queries. Rewards a relevant result appearing near the top; doesn't care how many relevant results exist in total, so it's a good proxy for "does the user find something useful quickly."
NDCG (Normalized Discounted Cumulative Gain). Rewards relevant results ranked higher with a logarithmic rank discount, normalized against the ideal ordering. With graded relevance labels (highly relevant / somewhat relevant / irrelevant) it's strictly more informative than MRR; the code below uses the simpler binary-relevance case, since that's what the substring-based ground truth actually supports.
Hit Rate@k. The coarsest of the five: for what fraction of queries was at least one relevant chunk found anywhere in the top-k? Easy to compute and communicate, but doesn't distinguish "one relevant chunk barely made it in" from "every retrieved chunk was relevant."
Building real ground truth from the data itself¶
The standard 8-question eval set used throughout this site carries a keyword field — a substring that only appears on the page(s) that actually answer that question. That's enough to build a real, verifiable relevance set per query, rather than hand-labeling "relevant" pages by eye:
def build_relevance_set(chunks: list, keyword: str) -> set:
"""Ground truth built from a real substring match, not hand-picked
page numbers: the relevant set for a question is every chunk whose
text actually contains the keyword that answers it."""
return {c["id"] for c in chunks if keyword.lower() in c["text"].lower()}
def retrieval_metrics(retrieved: list, relevant: set, k: int) -> dict:
"""Precision@k, Recall@k, Reciprocal Rank, NDCG@k (binary relevance),
and Hit Rate@k for one query. `retrieved` is a ranked list of chunk
ids; `relevant` is the real ground-truth relevant set for the query."""
import math
top_k = retrieved[:k]
hits_in_top_k = sum(1 for cid in top_k if cid in relevant)
precision = hits_in_top_k / k
recall = hits_in_top_k / len(relevant) if relevant else 0.0
reciprocal_rank = 0.0
for rank, cid in enumerate(retrieved, start=1):
if cid in relevant:
reciprocal_rank = 1.0 / rank
break
dcg = sum(1.0 / math.log2(rank + 1) for rank, cid in enumerate(top_k, start=1) if cid in relevant)
ideal_hits = min(len(relevant), k)
idcg = sum(1.0 / math.log2(rank + 1) for rank in range(1, ideal_hits + 1))
ndcg = dcg / idcg if idcg > 0 else 0.0
hit_rate = 1.0 if hits_in_top_k > 0 else 0.0
return {"precision": precision, "recall": recall, "reciprocal_rank": reciprocal_rank, "ndcg": ndcg, "hit_rate": hit_rate}
Measured across the standard 8-question eval set, dense retrieval at k=3:
One real miss explains most of the gap from a perfect score
The "How many attention heads did they use?" question scored 0.0 on Precision@3, Recall@3, and Reciprocal Rank. The chunk that actually contains "h = 8" sits on page 5 — but dense retrieval's top-3 for that query pulls in a page from the appendix's attention-visualization figures instead, a real but tangential semantic match that outranks the literal answer. This is the same retrieval miss the Vector Databases page's Qdrant and Weaviate demos hit on the identical question — a metadata filter fixed it there; here it shows up directly in the retrieval-metric numbers instead of being fixed.
Generation metrics: classic lexical overlap¶
These predate LLM-based evaluation, from machine translation and summarization research, and are worth knowing specifically for their limitations.
BLEU. Measures n-gram overlap between generated text and a reference — what fraction of generated word sequences (unigrams, bigrams, etc.) also appear in the reference. Originally built for machine translation.
ROUGE. Similar n-gram overlap, but recall-oriented — what fraction of the reference's n-grams appear in the generated text. Originally built for summarization. ROUGE-L specifically uses longest-common-subsequence overlap rather than fixed-size n-grams.
METEOR. Improves on BLEU by accounting for synonyms and stemming rather than requiring exact word matches, and balances precision and recall — generally more forgiving than raw BLEU on paraphrased but correct text.
Embedding-based semantic similarity. Embed both the generated answer and the reference, then compute cosine similarity — captures meaning-level closeness rather than surface wording, at the cost of not directly checking whether the answer is actually grounded in retrieved context.
def generation_metrics(reference: str, answer: str, embed_fn) -> dict:
"""Classic lexical-overlap metrics (BLEU, ROUGE-L, METEOR) plus
embedding-based semantic similarity, for one generated answer
against its reference. `embed_fn(text)` returns a normalized
embedding vector for cosine similarity via dot product."""
import sacrebleu
from nltk.translate.meteor_score import meteor_score
from rouge_score import rouge_scorer
bleu = sacrebleu.sentence_bleu(answer, [reference]).score
scorer = rouge_scorer.RougeScorer(["rougeL"], use_stemmer=True)
rouge_l = scorer.score(reference, answer)["rougeL"].fmeasure
meteor = meteor_score([reference.split()], answer.split())
ref_vec = embed_fn(reference)
ans_vec = embed_fn(answer)
semantic_similarity = float(ref_vec @ ans_vec)
return {"bleu": bleu, "rouge_l": rouge_l, "meteor": meteor, "semantic_similarity": semantic_similarity}
Measured against 8 real answers generated locally (no paid API):
A factually wrong answer that still scores well on every metric here
Asked "What BLEU score did they get on English-to-German translation?", the naive RAG pipeline answered "...their big model achieved a BLEU score of 41.0 on the WMT 2014 English-to-German translation task" — factually wrong; 41.0 is the paper's English-to-French score, not English-to-German (28.4). That wrong answer still scored BLEU=32.9, ROUGE-L=0.68, METEOR=0.76, semantic similarity=0.89 — all comfortably high, because the wrong number sits inside an otherwise well-formed, structurally similar sentence. None of these four metrics has any mechanism for checking whether the specific number is correct; they all measure closeness in wording or meaning to the reference, not truth. This is the concrete version of the abstract warning every metrics tutorial gives about BLEU/ROUGE: a factually wrong answer phrased similarly to the reference can score just as well as a right one.
Beyond lexical overlap: groundedness, completeness, utilization¶
A RAG-specific framing, complementary to the metrics above:
- Groundedness (faithfulness): is every claim in the answer actually supported by retrieved context? The core hallucination check — an unsupported claim is either drawn from the model's own parametric memory (risky) or fabricated outright.
- Completeness: does the answer use all the relevant information available in retrieved context, or leave out something important that was actually retrieved? Groundedness alone doesn't catch this — an answer can be perfectly faithful and still incomplete.
- Utilization: of everything retrieved, how much actually shows up in the generated answer? Low utilization can mean the model is ignoring good context (a generation-side problem) or that retrieval brought back irrelevant padding the model correctly ignored — which one it is matters more than the raw number.
Checking these properly needs an LLM judge — decomposing an answer into individual claims and checking each one against retrieved context, the same way BLEU/ROUGE can't. That's exactly what RAGAS's Faithfulness and Context Recall metrics do, and RAGAS Deep Dive has real, measured numbers from running them against this same corpus — including a case where a factually correct answer still scored near-zero on groundedness, the mirror image of this page's BLEU/ROUGE finding above.
Human evaluation¶
Every automated metric on this page and on RAGAS Deep Dive is a proxy for what actually matters: a correct, useful answer to a real question. A response can look fine on every automated number while still being subtly unhelpful, tonally wrong, or wrong in a way none of these metrics were built to catch.
Direct scoring. Raters score responses on defined dimensions (correctness, helpfulness, groundedness) against a written rubric, for consistency across raters.
Pairwise comparison. Show raters two candidate responses and ask which is better — often more reliable than absolute scoring, since relative judgments are cognitively easier than absolute ones.
Inter-rater agreement. With multiple raters, measuring agreement (e.g. Cohen's kappa) surfaces whether a rubric is ambiguous or the task is genuinely subjective — low agreement means the resulting scores need lower confidence attached to them.
In practice, human evaluation is expensive enough that it's usually reserved for validating that an automated metric (RAGAS, an LLM judge, or the metrics on this page) actually correlates with human judgment on a sample, plus periodic spot-checks of production output, since automated judges can drift or carry blind spots that only surface over time.
Scenario Check¶
Sources & further reading¶
- sacrebleu — the BLEU implementation used above
- rouge-score — Google's ROUGE implementation used above
- NLTK METEOR — the METEOR implementation used above
- RAGAS Deep Dive — the LLM-judged groundedness/faithfulness diagnostic table this page's classic metrics complement, with real measured numbers on the same corpus
- Vector Databases — the same retrieval miss this page's Precision@3=0 example shows up in directly, fixed there with a metadata filter