Skip to content

Response Quality & Safety

RAG systems fail in ways plain chatbots don't, and get evaluated on axes plain chatbots don't need: whether an answer is actually grounded in what was retrieved, whether a document silently smuggled in an instruction the model shouldn't follow, and whether a chunk that belongs to one user leaks into another user's answer. This page covers hallucination mitigation, guardrails, RAG-specific prompt injection, and output/PII safety — current tools and current incidents, not a generic AI-safety checklist.

TL;DR — quick revision
  • Aggregate faithfulness scores hide per-claim hallucination — a response with 5 claims and 1 fabricated detail can still average ~0.8 on a faithfulness metric; the fix is claim-level decomposition (what RAGAS's Faithfulness metric already does), not a single answer-level verdict
  • Anthropic's Citations API grounds attribution, not truth — it guarantees a claim came from an exact source passage, not that the claim is correct; a 2025 OpenAI/Georgia Tech paper argues hallucination persists because training-time benchmarks reward confident guessing over calibrated "I don't know" answers
  • RAG's prompt-injection surface is the retrieved content itself, not just the user's message — EchoLeak (CVE-2025-32711) was a real, zero-click production exploit where a hidden instruction in an unopened email got pulled into Microsoft 365 Copilot's context and exfiltrated data
  • Retrieval-time access control must be a hard metadata filter at the database layer, not a prompt instruction — the LLM cannot be relied on as a security boundary for content it was never supposed to see in the first place

Hallucination mitigation in RAG

Giving a model correct retrieved context doesn't guarantee it uses that context faithfully. It can still ignore correct context and answer from parametric memory, blend a real retrieved fact with a fabricated detail in the same sentence, or misattribute a claim to the wrong source. RAG lowers hallucination risk by grounding generation in retrieved evidence — it doesn't eliminate it.

Aggregate metrics create a false sense of safety. A response with five claims and one fabricated detail still scores roughly 0.8 on an averaged, answer-level faithfulness check — a dashboard can show 92% aggregate faithfulness while the true per-claim hallucination rate sits around 18%. Current research on this points toward claim-level decomposition rather than a single yes/no verdict per answer, which is exactly how RAGAS's Faithfulness metric works: decompose the response into atomic claims, check each one individually against retrieved context, and score supported-claims ÷ total-claims — a graded signal instead of one binary judgment that loses which specific claim is the problem. A newer variant, FaithfulnessWithHHEM, swaps the LLM-as-judge step for Vectara's HHEM-2.1-Open — a purpose-built entailment classifier — specifically to avoid using an LLM to judge another LLM's output.

Citation and attribution. Anthropic's Citations API (GA since mid-2025, also available on AWS Bedrock and other platforms) lets a model return character-level spans pointing to the exact source sentence behind each claim in its answer. The distinction worth being precise about in an interview: this grounds attribution, not truth — it guarantees "this text came from this exact passage," not "this claim is correct." A model can still misquote or misinterpret a passage it faithfully cites, or cite a passage that only partially supports the specific claim next to it. It's also retrieval-agnostic: the application still has to do retrieval itself; Citations only handles the "show your work" step afterward.

Uncertainty-based detection. The original semantic entropy method detects hallucination by sampling many generations for the same query, clustering them by meaning, and measuring entropy across the clusters — high semantic entropy means the model is confabulating rather than converging on an answer. That's expensive (5–10× the inference cost for the extra samples). Semantic Entropy Probes (Kossen et al., 2024) train a cheap linear probe directly on a single generation's hidden states to approximate the same signal without the sampling overhead. A 2025 RAG-specific extension, SEReDeEP, fuses this kind of entropy signal with a second signal distinguishing context-usage from parametric-memory-usage — closer to directly detecting the "ignored good context and answered from memory instead" failure mode this site has measured directly in RAGAS Deep Dive.

The root-cause argument: it's a training-incentive problem, not just a missing detector

A September 2025 paper from OpenAI and Georgia Tech, "Why Language Models Hallucinate", argues hallucinations are ordinary binary classification errors: if a model can't statistically distinguish a true statement from a false one during pretraining, producing false statements is the expected consequence, not a mysterious failure. Its sharper claim is that hallucination persists after training because the benchmarks models are optimized against reward confident guessing over admitting uncertainty — like a multiple-choice exam where guessing beats leaving an answer blank. The paper's proposed fix is to change how existing benchmarks are scored (penalize confident wrong answers more than calibrated "I don't know," rather than inventing another hallucination leaderboard) — a training-time, incentive-design fix, in contrast to every other technique on this page, which operates at inference time.

Guardrails: current tools and mechanics

Guardrails work best as checkpoints at each pipeline stage, not one blanket filter: input (validate the query before retrieval — off-topic detection, PII, injection attempts), retrieval (access-control filtering as a hard metadata constraint, covered in detail below), output (toxicity, PII leakage, policy violations before a response reaches the user), and tool/action (for agentic RAG — human-in-the-loop confirmation for anything irreversible).

NVIDIA NeMo Guardrails (docs, GitHub) defines programmable "rails" — input, dialogue, retrieval, output, and execution — using Colang, a small domain-specific language, so a policy like "never discuss competitor pricing" or "reject retrieved content matching an injection pattern" is expressed as a flow the framework checks around every LLM call, rather than baked into the prompt itself.

Meta's Llama Guard 4 (model card) is the current generation — a 12B multimodal safety classifier that scores both prompts (before the main LLM sees them) and generated responses (after) against a 14-category harm taxonomy, with per-category opt-out. It unified what used to be two separate text- and vision-specific Llama Guard 3 models. Prompt Guard 2 (model card), a much smaller 86M (or 22M, for lower latency) classifier, is specifically for detecting jailbreak/injection-style text — small enough to run as a pre-filter over every retrieved chunk, not just the user's own message, which is exactly the RAG-relevant use case. Code Shield, the third piece of Meta's PurpleLlama toolkit, screens LLM-generated code for insecure patterns before it's executed or shown, relevant wherever a RAG system's output feeds into code execution.

Guardrails AI (docs) wraps LLM calls in a Guard object with attached Validators (PII detection, toxicity, jailbreak detection, and others) that can fix, reask, filter, or hard-fail on a violation. It's mid-migration away from its hosted Hub toward plain pip-installable validator packages, with the old hosted-inference path fully sunsetting August 25, 2026 — worth checking which install path a given tutorial assumes before following it.

Spotlighting: tagging retrieved content as lower-trust, not filtering it out

Microsoft's Azure AI Content Safety ships Prompt Shields, which separately detects direct attacks (a user trying to override system instructions) and document attacks — hidden instructions embedded in third-party content like retrieved documents, emails, or tool output, scanned at both the input and tool-response stage. Its Spotlighting feature (preview) is the piece most specific to RAG: it transforms retrieved document content into base64 before it reaches the model, which marks that content as structurally distinct from direct instructions and makes embedded commands harder to parse as authoritative. The trade-offs are real: base64 inflates token count (cost, and can blow context limits on long documents), it's Chat-Completions-API-only, off by default, and the model occasionally starts narrating that content is base64-encoded in its own answers. Docs →

OpenAI's Moderation API is now solely omni-moderation-latest, a GPT-4o-based multimodal (text + image) classifier, free to call — the older text-only moderation model was retired in late 2025.

A guardrail layered around a model, rather than baked into its weights, can be independently measured and cost-optimized the same way any other production system can — Anthropic's own Constitutional Classifiers went through two full generations in production, cutting compute overhead roughly 40× while cutting false-positive refusals 87%, with real red-teaming numbers behind both claims.

Prompt injection: the RAG-specific angle

In a plain chatbot, the only attacker-controlled channel into the model's context is the user's own message — an attacker has no incentive to inject something malicious into a prompt only they will read. RAG changes this: part of the model's context comes from documents the current user never wrote and may not have chosen — search results, scraped pages, other users' shared documents, tool output. Any of those can carry a natural-language instruction planted by an attacker, and unless the pipeline explicitly marks that content as lower-trust (Spotlighting, provenance tracking, or a structural defense like CaMeL below), the model has no built-in way to tell it apart from a legitimate instruction.

This isn't theoretical. The InjecAgent benchmark (ACL 2024), testing tool-using agents against indirect injection delivered through tool output, found ReAct-prompted GPT-4 vulnerable 24% of the time — and higher when the injected instruction used simple reinforcement tricks like repetition.

EchoLeak: the first documented zero-click indirect injection in a production system

CVE-2025-32711, CVSS 9.3 — privately reported to Microsoft in January 2025, patched server-side in May 2025, and publicly disclosed June 11, 2025. An attacker sends a victim a single, unsolicited email containing a prompt-injection payload phrased as ordinary business prose — the victim never has to open or interact with it. Later, when the victim asks Microsoft 365 Copilot an unrelated question, Copilot's own RAG retrieval pulls the malicious email into context because it looks semantically relevant, and the model executes the embedded instructions. The exploit chained four separate bypasses to achieve zero-click exfiltration — evading Microsoft's cross-prompt-injection classifier, defeating link redaction via reference-style Markdown, using an auto-fetched image tag instead of a clickable link, and routing through Microsoft's own CSP-allowlisted Teams proxy — a pattern researchers termed "LLM scope violation": untrusted data influencing behavior beyond the scope it should be confined to. It's the first documented real-world (not red-team) case of RAG-delivered indirect injection achieving actual data exfiltration. Full case study →

CaMeL (Google DeepMind, 2025) is a structural answer to the same problem, not another prompt-level mitigation. It splits the agent into a Privileged LLM that only ever sees the trusted user query and produces a plan as restricted, code-like structure (not free text), and a Quarantined LLM with no tool-calling capability at all, whose only job is extracting structured values out of untrusted data when the plan calls for it. A custom interpreter attaches provenance metadata ("capabilities") to every value as it flows through the plan, and enforces policies like "never call send_email if the recipient's value ever touched untrusted data" — regardless of what the generated code appears to want to do. On the AgentDojo benchmark, CaMeL solved 77% of tasks with provable security guarantees versus 84% for an undefended baseline — a real but bounded capability cost for a structural, not probabilistic, defense.

Where RAG risk sits in OWASP's current taxonomy. The OWASP Top 10 for LLM Applications (2025) gives RAG-relevant risk two dedicated categories that didn't exist in the prior list: LLM01 Prompt Injection (explicitly naming indirect injection via retrieved content as a subtype), LLM02 Sensitive Information Disclosure, and LLM08 Vector and Embedding Weaknesses — covering embedding inversion, cross-tenant retrieval leakage, and corpus poisoning. LLM08 being new is itself a signal: OWASP's own view is that RAG's data layer earned a category of its own.

Output safety and PII in RAG

The RAG-specific version of a PII leak doesn't require the current user's query to contain any PII at all — the leak originates from the retrieved side. A chunk belonging to a different customer or tenant can end up in context, and unless retrieval is permission-filtered, the model will synthesize it into an answer for the wrong person just as readily as it would any other retrieved fact.

The structurally correct fix is retrieval-time access control, enforced as a hard metadata filter in the vector search call itself — not a prompt instruction telling the model to "be careful," which relies on the LLM as a security boundary it was never designed to be. Each chunk is tagged at ingestion with metadata mirroring the source system's real ACLs (allowed users, groups, tenant ID), and that metadata filters the similarity search before results ever reach the model — the same metadata-filtering mechanism covered in depth on the Vector Databases page, and only as strong as the isolation guarantees of whichever multi-tenancy pattern (namespace-per-tenant, shared index with filtering, or index-per-tenant) that page describes. In practice this is necessary but not sufficient: filters fail silently when a source system's ACLs are stale or incompletely synced into vector metadata, which is why some architectures add a second layer — entity-level re-verification that strips over-permissioned content even after a chunk clears the coarse document-level filter — and choose between pre-filtering (apply the ACL filter before the similarity search, efficient when authorized content is a small fraction of the corpus) and post-filtering (search first, drop unauthorized hits after, fine when it isn't).

Output-side scanning is defense-in-depth, not a substitute for the above. Microsoft Presidio (open source) runs NER-plus-regex detection and redaction/masking, and the recommended pattern runs it at three points — on the incoming query, on each retrieved chunk before it enters the prompt, and again on the model's final output — rather than trusting a single checkpoint. Presidio's own documentation is explicit that "there is no guarantee Presidio will find all sensitive information," which is exactly why it sits alongside retrieval-time ACLs rather than replacing them. Commercial alternatives (Nightfall AI, AWS Comprehend PII detection, Google Cloud DLP) offer the same detect-and-redact pattern as a managed API, increasingly integrated at the agent-gateway level rather than called ad hoc.

Scenario Check

Sources & further reading