Skip to content

CAG (Cache-Augmented Generation)

Every technique so far retrieves something per query — search a knowledge base, pull the relevant chunks, feed them into the prompt. CAG questions that assumption for a specific situation: if the entire knowledge base is small enough to fit in the model's context and doesn't change often, skip retrieval entirely. Load the whole thing into context, reuse the expensive part of processing it across every subsequent query, and let the model answer with the full document always "in mind" instead of whatever a similarity search happened to surface.

Scope: what this page actually measures

The original CAG paper's mechanism is precomputing and reusing a model's raw KV-cache — only possible when you control the model's internals, which means a self-hosted, open-weight model. Hosted APIs (Claude, OpenAI) never expose that internal state for you to save and reload, and an 8B+ open-weight model isn't practical to run locally without a dedicated GPU. This page measures the closest real equivalent available via a hosted API: Anthropic's cache_control prompt caching, applied to an entire document as the system prompt. It's a genuine, honest substitution for the paper's actual mechanism, not the literal thing — see below for the exact distinction.

Hands-on

The full comparison below is runnable, with real API calls against the same sample paper used across this site: CAG → in the code repo. Needs an Anthropic API key; there's no local/free path here since the whole technique is about a hosted model's context and caching behavior.

TL;DR — quick revision
  • No retrieval at all: the entire document goes into the system prompt on every call — there's no embedding step, no vector search, no chunk selection anywhere in this pipeline
  • "Fresh" (plain long-context stuffing) vs. "cached" (CAG's actual claim) are different things: sending the whole document every call is not CAG by itself — what makes it CAG is reusing the same processed prefix across queries instead of reprocessing it every time
  • Cost reduction is dramatic and consistent, measured directly: cache reads are billed at 0.1x the base input price vs. 1.25x to write the cache and 1.0x with no caching at all — reprocessing a ~15K-token document on every call costs roughly 6-7x what reading it from cache costs, measured at 84-86% total cost reduction across two full runs
  • Latency reduction was NOT reliably observed at this document size: the measured delta swung from +13% (cached actually slower) to -5% (cached faster) across two runs — noise-level, not a dependable win, even though Anthropic's own published figures claim up to 85% latency reduction at larger scale
  • Caching can't change accuracy, and it didn't: the model reads byte-identical content either way, so any answer differences between runs are ordinary LLM phrasing variance, not a caching effect
  • It doesn't scale by using an even bigger context window — a documented hybrid pattern ("RAG over CAG") exists for multi-document knowledge bases, with its own real trade-offs, covered below

Fresh vs. cached: the distinction that actually matters

Fresh — the entire document is sent as the system prompt on every single call, with no caching. This alone is not CAG; it's just long-context stuffing, and it's expensive: the model reprocesses the full document from scratch every time, at full price, regardless of whether the document changed since the last call.

Cached — the same system prompt, but annotated with a cache breakpoint. The first call against a given document pays a real premium to write the cache. Every later call within the cache's TTL, against the byte-identical prefix, reads from the cache instead of reprocessing the document — this reuse across repeated queries is the entire mechanism CAG's name refers to. Conflating "long context" with "caching" is the most common mix-up in casual coverage of this topic: the size of what you put in context and whether that context gets reused are two separate levers.

def load_document_text(pdf_path: str) -> str:
    """The entire document, not chunks of it -- CAG's whole premise is
    that the corpus is small enough to load in full."""
    from pypdf import PdfReader

    reader = PdfReader(pdf_path)
    return "\n\n".join(page.extract_text() or "" for page in reader.pages)

def generate_fresh(document_text: str, question: str, model: str) -> tuple:
    """Plain long-context stuffing: the entire document is sent as the
    system prompt on every call, with no cache_control at all. This is
    the baseline CAG's caching layer is compared against -- not RAG, but
    the "just put everything in context, every time" alternative."""
    import time

    import anthropic

    client = anthropic.Anthropic()
    start = time.perf_counter()
    response = client.messages.create(
        model=model,
        max_tokens=1024,
        system=document_text,
        messages=[{"role": "user", "content": question}],
    )
    elapsed = time.perf_counter() - start

    answer = next(block.text for block in response.content if block.type == "text")
    usage = {
        "input_tokens": response.usage.input_tokens,
        "cache_creation_input_tokens": getattr(response.usage, "cache_creation_input_tokens", 0) or 0,
        "cache_read_input_tokens": getattr(response.usage, "cache_read_input_tokens", 0) or 0,
        "output_tokens": response.usage.output_tokens,
    }
    return answer, usage, elapsed

def generate_cached(document_text: str, question: str, model: str, ttl: str = "5m") -> tuple:
    """Same document, same question shape, but the system prompt carries a
    `cache_control` breakpoint. The FIRST call against a given document
    writes the cache (billed at a real premium: 1.25x base input price for
    a 5-minute TTL). Every subsequent call within the TTL, against the same
    exact prefix, reads from the cache instead of reprocessing it (billed
    at 0.1x base input price)."""
    import time

    import anthropic

    client = anthropic.Anthropic()
    start = time.perf_counter()
    response = client.messages.create(
        model=model,
        max_tokens=1024,
        system=[{"type": "text", "text": document_text, "cache_control": {"type": "ephemeral", "ttl": ttl}}],
        messages=[{"role": "user", "content": question}],
    )
    elapsed = time.perf_counter() - start

    answer = next(block.text for block in response.content if block.type == "text")
    usage = {
        "input_tokens": response.usage.input_tokens,
        "cache_creation_input_tokens": getattr(response.usage, "cache_creation_input_tokens", 0) or 0,
        "cache_read_input_tokens": getattr(response.usage, "cache_read_input_tokens", 0) or 0,
        "output_tokens": response.usage.output_tokens,
    }
    return answer, usage, elapsed

Cost isn't a single flat "cached" rate — cache writes and cache reads are both multipliers on Sonnet 5's base input price ($2.00/MTok), computed directly from the API's own usage fields on each response:

def cost_from_usage(usage: dict) -> float:
    """Real dollar cost for one call, computed from its actual usage
    fields and Sonnet 5's published per-token rates -- not an estimate.
    Cache write and cache read are MULTIPLIERS on the base input price,
    not separate flat rates."""
    INPUT_PRICE_PER_MTOK = 2.00
    OUTPUT_PRICE_PER_MTOK = 10.00
    CACHE_WRITE_5M_MULTIPLIER = 1.25
    CACHE_READ_MULTIPLIER = 0.1

    input_cost = usage["input_tokens"] / 1_000_000 * INPUT_PRICE_PER_MTOK
    cache_write_cost = usage["cache_creation_input_tokens"] / 1_000_000 * INPUT_PRICE_PER_MTOK * CACHE_WRITE_5M_MULTIPLIER
    cache_read_cost = usage["cache_read_input_tokens"] / 1_000_000 * INPUT_PRICE_PER_MTOK * CACHE_READ_MULTIPLIER
    output_cost = usage["output_tokens"] / 1_000_000 * OUTPUT_PRICE_PER_MTOK
    return input_cost + cache_write_cost + cache_read_cost + output_cost

Measured: cost drops sharply, latency doesn't — reliably

Two independent full runs, same document (~15,125 tokens), same 8 questions used across this repo:

                  Fresh (total)   Cached (total)   Change
Cost, run 1       $0.2536         $0.0399          -84.3%
Cost, run 2       $0.2544         $0.0370          -85.5%
Latency, run 1    27.96s          31.63s           +13.1% (slower)
Latency, run 2    28.30s          26.93s            -4.8% (faster)

Cost: dramatic, consistent, and easy to explain

Cache reads are billed at 0.1x the base input price; a fresh call costs 1.0x; the initial cache write costs 1.25x. Reprocessing a ~15K-token document on every call is genuinely expensive compared to reading it back from cache — this shows up as an 84-86% total cost reduction both times it was measured, a real and reliable effect at this document size.

Latency: not a reliable win here, despite the vendor's own marketing claim

Anthropic publishes "up to 85% latency reduction" for prompt caching. Measured directly across two full runs here, the actual delta ranged from cached being 13% slower to cached being 5% faster — essentially noise, not a dependable improvement. The likely reason: at ~15,000 tokens, network round-trip time and output generation dominate total wall-clock time far more than the prefill compute that caching actually saves. The vendor's claim almost certainly holds at a much larger cached-context size — hundreds of thousands of tokens, where prefill compute is a bigger share of total request time — but it doesn't reproduce at the scale this recipe's sample document sits at. Cost savings and latency savings are genuinely separate claims, and this page's own numbers show them diverging.

Accuracy was identical either way, as expected — 6-7 of 8 correct on both fresh and cached runs. Caching doesn't change what content the model reads, only how the unchanged prefix is billed and computed, so there's no mechanism by which it could affect correctness. The one recurring miss on this repo's standard eval set is a keyword-matching artifact, not a real error: the model correctly answers d_model = 512, but the eval's exact-match keyword is dmodel = 512 (no underscore) — a formatting difference, not a wrong answer.

Scaling beyond one document: "RAG over CAG"

CAG doesn't scale by reaching for an ever-bigger context window — computing one cached prefix over many documents together stops being practical well before you'd want it to, and a knowledge base that's actually large and growing needs something else. A documented hybrid pattern handles this: cache each document separately, then use a lightweight router — an embedding comparison or a small classifier, the same routing idea covered on the Modular RAG page, just applied to picking a cache instead of a retrieval path — to decide which document's precomputed cache to load for a given query. On a router miss, fall back to ordinary retrieval rather than forcing a wrong cache.

Concretely: a company with 5 separate policy handbooks could precompute and store one cache per handbook offline, keep a small per-handbook description or embedding in a lightweight index, and route each incoming question to the one handbook's cache that actually matches it — no retrieval within a handbook once it's selected, since the whole document is already in context.

This has a real, honest limitation: a question that genuinely needs two handbooks at once forces the router to either pick one (an incomplete answer) or load both caches together, which may not fit and isn't a cleanly solved problem in the literature this pattern comes from. The router is also a single point of failure in a way pure retrieval isn't — a misrouted query loads the wrong document's cache entirely, with none of the partial credit that fusing multiple retrieved sources can provide.

When to use which

RAG CAG
Knowledge base size Scales to large, growing corpora Limited to what fits in context
Cost per repeated query Retrieval cost + a smaller generation context every time Near-zero marginal cost once cached, measured directly above
Latency per repeated query Retrieval latency + generation Not reliably faster at moderate document sizes (measured above) — the win is compute-share-dependent, not automatic
Updates to the knowledge base Re-embed/re-index just the changed document Cache invalidates on any byte change to the cached prefix — no incremental update path
Retrieval failure risk Real (wrong chunk retrieved, missed context) None — the whole document is always present

Use CAG for: a small-to-medium, relatively static knowledge base where retrieval errors are a bigger practical risk than the cost of holding it all in context — a single detailed product manual, one contract analyzed repeatedly, a static internal FAQ. Use RAG for: a genuinely large or fast-changing corpus, or workloads where per-query latency specifically (not just cost) needs to improve — this page's own measurement shows caching doesn't automatically deliver that. Use "RAG over CAG" for: a knowledge base made of several distinct, individually-small documents, where routing to the right one is easier than merging them into a single cached context.

Scenario Check

Sources & further reading