Skip to content

Production Considerations

A RAG pipeline that works in a notebook and one that survives production traffic differ in specific, learnable ways: what happens when a dependency starts failing, what actually costs money at scale, what a generic uptime dashboard can't see, and where throughput actually breaks. This page covers all four with real, tested code for the reliability and cost/monitoring mechanics, and current, sourced numbers for the parts that are inherently about infrastructure scale rather than something to demo on a laptop.

Hands-on

The reliability, latency, cost, and caching demos below are runnable: Production Considerations → in the code repo. No paid API needed — generation runs through a local Ollama model.

TL;DR — quick revision
  • Retries, fallbacks, and circuit breakers are three different tools, not one — retries handle transient failures, fallbacks route to a secondary provider, circuit breakers proactively stop sending requests to something that's already failing; this page's own circuit-breaker demo shows the real latency saved by not retrying a doomed call
  • Semantic caching and prompt caching solve different problems — semantic caching skips the whole pipeline for a near-duplicate query (this page measured a real 33% hit rate, with a genuine miss right at the similarity threshold boundary); prompt caching (Anthropic's, mechanically confirmed) caches a shared prefix within calls that still each generate fresh output
  • Standard uptime monitoring cannot see RAG's actual failure mode — a system serving wrong answers stays fast and returns 200 OK on every request, so retrieval-quality drift needs its own monitoring, not a byproduct of latency/error-rate dashboards
  • Scalability at real production numbers is a GPU-throughput and region-placement problem for embedding, and a right-sized-model problem for every auxiliary LLM call — with real published numbers for both, not estimates

Reliability: retries, fallbacks, and circuit breakers are different tools

It's tempting to treat these as one "handle failures" bucket, but they solve different problems and combine into a layered defense, not a single fallback path.

Retries exist for transient failures — a brief rate limit, a cold start, a dropped connection — where trying again with backoff is likely to succeed. tenacity is the standard current Python library for this:

def demo_retry_backoff() -> str:
    """Retries a failing call with exponential backoff via `tenacity`.
    `retry_if_exception_type` only retries the specific exception a
    transient failure would raise; `stop_after_attempt` bounds the
    worst-case cost of a call that never recovers."""
    from tenacity import retry, retry_if_exception_type, stop_after_attempt, wait_exponential

    attempts = {"count": 0}

    def flaky_call() -> str:
        attempts["count"] += 1
        if attempts["count"] <= 2:  # fails the first two times, then succeeds
            raise ConnectionError(f"reranker service unavailable (attempt {attempts['count']})")
        return f"reranked results (succeeded on attempt {attempts['count']})"

    @retry(
        retry=retry_if_exception_type(ConnectionError),
        wait=wait_exponential(multiplier=0.2, min=0.2, max=2),
        stop=stop_after_attempt(5),
        reraise=True,
    )
    def call_with_retry() -> str:
        return flaky_call()

    return call_with_retry()

Measured against a call that fails twice, then succeeds:

attempt 1: failed: reranker service unavailable (attempt 1)
attempt 2: failed: reranker service unavailable (attempt 2)
attempt 3: success

Retrying blindly has a real cost: too many retries on a call that's failing for a non-transient reason just adds latency and can trigger a "retry storm" that makes an already-struggling dependency worse.

Circuit breakers exist for the opposite case — a dependency that's failing persistently, where retrying is pure waste. Once a failure threshold is crossed, the breaker opens and short-circuits further calls immediately, with no request sent at all, until a cooldown period passes and it allows a single probe request through to check recovery:

def demo_circuit_breaker(num_calls: int = 6) -> list[str]:
    """Wraps a persistently failing call in a `pybreaker` circuit
    breaker. After `fail_max` consecutive failures the breaker opens
    and short-circuits further calls immediately -- no request is sent
    at all -- until `reset_timeout` elapses."""
    import time

    import pybreaker

    breaker = pybreaker.CircuitBreaker(fail_max=3, reset_timeout=2)

    @breaker
    def call_down_service() -> None:
        time.sleep(0.05)  # simulates real request latency before the failure is known
        raise ConnectionError("reranker service is down")

    log = []
    for i in range(1, num_calls + 1):
        start = time.monotonic()
        try:
            call_down_service()
        except pybreaker.CircuitBreakerError:
            log.append(f"call {i}: short-circuited immediately ({time.monotonic() - start:.4f}s, no request sent)")
        except ConnectionError:
            log.append(f"call {i}: request sent and failed ({time.monotonic() - start:.4f}s)")
    return log

Measured against a persistently down service, 6 calls:

call 1: request sent and failed (0.0538s)
call 2: request sent and failed (0.0509s)
call 3: short-circuited immediately (0.0557s, no request sent)
call 4: short-circuited immediately (0.0001s, no request sent)
call 5: short-circuited immediately (0.0000s, no request sent)
call 6: short-circuited immediately (0.0000s, no request sent)

Only 3 of the 6 calls actually reached the service — the breaker opened after the third failure and every subsequent call returned instantly instead of waiting on a doomed request. That's the real, measured value: not just fewer errors, but genuinely avoided latency.

Fallbacks are a third, separate tool — routing to a secondary provider or model when the primary fails, rather than retrying the same one or giving up. Worth knowing that a fallback isn't automatically independent risk: a primary and secondary that share the same upstream infrastructure can fail together, so a fallback is only as good as its actual failure-domain independence from the primary.

LLM-specific reasons this space needs more care than generic microservice circuit-breaker theory: LLM providers have rate limits that can trigger mid-conversation without warning, latency variance that can span 800ms to 15+ seconds on the same endpoint, and a failure mode generic breakers don't model at all — a 200 OK response that's silently lower-quality under peak load, which no HTTP-status-based breaker would ever trip on.

Circuit-breaker state needs to be shared across workers

pybreaker's in-memory state is thread-safe but per-process — running multiple worker processes (the normal case in production) means each one tracks failures independently unless the breaker's state is backed by something shared (Redis is the common choice). A breaker that only sees a fraction of total traffic per process opens far later than intended.

Graceful degradation when retrieval itself fails

Retry and circuit-breaker logic assumes the call can fail. A RAG-specific question is what happens when retrieval succeeds mechanically but returns nothing useful, or the vector DB is unreachable entirely. Corrective RAG (CRAG) is the closest thing to a real, research-grade answer to this: it explicitly scores retrieved documents for relevance after retrieval, and triggers a live web search as the correction step when they're judged irrelevant — a genuine fallback ladder, not just a retry of the same failing retriever.

The anti-pattern worth naming explicitly: caching one generic "sorry, something went wrong" response as the fallback for every degraded query. Every degraded request then returns identical, unhelpful text regardless of what was actually asked — users notice this immediately, and it's strictly worse than a narrower, query-aware degradation path (a secondary keyword-based retriever, a curated FAQ fallback, or an honest "I don't have enough information" that at least engages with the actual question).

Cost management: two different caching levers

Prompt caching (Anthropic's implementation, confirmed directly from current docs) caches a shared prefix of a request — the parts of a prompt (a system prompt, static instructions, a stable block of retrieved context) that stay identical across calls — so only the actually-varying suffix needs fresh processing. Current mechanics: a cache write costs 1.25× the base input-token price for a 5-minute TTL, or 2× for a 1-hour TTL; a cache read costs roughly 0.1× the base input price (a 90% discount) on most current models. The cache clock resets on every read, so a steady stream of requests keeps an entry alive indefinitely without repaying the write cost. There's a minimum cacheable prefix length (1,024 tokens on Sonnet-class models, varying by model tier) — below it, a cache directive is silently ignored, no error. This maps directly onto RAG's repeated-system-prompt-plus-retrieved-context pattern: cache everything up to the final user turn, and only that turn needs fresh processing on every call.

Semantic caching solves a different problem: skipping the entire pipeline — embedding, retrieval, and generation — for a query that's a near-duplicate of one already answered, not just reusing a prefix within a call that still generates fresh output. Real current tools (GPTCache, Redis's semantic caching) work the same way mechanically: embed the incoming query, check cosine similarity against previously-cached query embeddings, and serve the cached response directly on a hit above some similarity threshold.

def semantic_cache_lookup(query_vec, cache: list, threshold: float = 0.92):
    """Checks a query embedding against a cache of past query
    embeddings; returns the cached answer above `threshold` cosine
    similarity, or None on a miss. `cache` is a list of
    {"query_vec", "question", "answer"} dicts, mutated in place by the
    caller on a miss."""
    best_sim, best_entry = 0.0, None
    for entry in cache:
        sim = float(query_vec @ entry["query_vec"])
        if sim > best_sim:
            best_sim, best_entry = sim, entry

    if best_entry is not None and best_sim >= threshold:
        return best_entry["answer"], best_sim
    return None, best_sim

Measured against six real questions, three of them genuine paraphrases of earlier ones, at a 0.85 similarity threshold:

CACHE MISS (best_sim=0.000)  'How many attention heads did they use?'
CACHE MISS (best_sim=0.126)  "What is the model's embedding dimension?"
CACHE HIT  (sim=0.857)  'How many attention heads does the model have?'  ->  matched 'How many attention heads did they use?'
CACHE MISS (best_sim=0.286)  'What optimizer was used for training?'
CACHE HIT  (sim=0.864)  "What's the embedding size of the model?"  ->  matched "What is the model's embedding dimension?"
CACHE MISS (best_sim=0.826)  'What optimizer did they use to train the model?'

2 hits, 4 misses out of 6 queries (33% hit rate)

The threshold is a real precision/recall trade-off, not a formality

At a stricter 0.92 threshold, all three genuine paraphrases missed — even "What's the embedding size of the model?" against "What is the model's embedding dimension?" only reached 0.864 cosine similarity with all-MiniLM-L6-v2 embeddings, comfortably below 0.92. Lowering to 0.85 caught two of the three real paraphrases, but the third — "What optimizer did they use to train the model?" against "What optimizer was used for training?" — still missed at 0.826, right at the boundary. This is the real trade-off semantic caching guides consistently point at: too high a threshold and genuinely repeated questions still cost a full pipeline run; too low and a cache hit can serve a wrong answer to a question that only looks similar. Current guidance from Redis's own semantic-caching documentation frames a false-positive-hit rate above roughly 3–5% as a signal to fix the underlying architecture, not just tighten the threshold further.

Cost drivers worth naming directly, since they set the priority order for where to actually spend optimization effort: LLM generation is consistently the largest ongoing cost line in a production RAG system, vector DB hosting is usually the smallest, and embedding-generation cost is a function of tokens processed × GPU cost per hour ÷ throughput — meaning it scales with corpus size and update frequency, not query volume.

Monitoring: what standard uptime dashboards cannot see

A RAG system that hallucinates, retrieves the wrong chunks, or silently degrades under load still returns a fast, 200 OK response on every request. Latency, error rate, and throughput — the metrics a generic API monitoring setup already tracks — capture none of the failure modes that actually matter for answer quality. This is the single most important framing for this section: retrieval-quality monitoring has to be built deliberately, not inherited for free from infrastructure monitoring.

Current tools built specifically for this (not generic APM): Arize Phoenix traces a RAG pipeline as a span tree with dedicated span types for embedding, retrieval (including the actual retrieved chunks and their similarity scores, not just a "retrieval happened" log line), and generation — and runs LLM-assisted evals at both the per-chunk level (a precision@k-style relevance judgment) and the full-answer level. Langfuse and LangSmith offer similar full trace trees with token usage, latency, and cost per span, plus the ability to build eval datasets directly from real production traffic rather than only a static eval set. Helicone takes a lighter-weight, proxy-based approach — no code/SDK change beyond swapping a base URL — trading pipeline-level diagnostic depth for much lower integration effort; it's a better fit for fast cost/usage tracking than for diagnosing exactly where a RAG pipeline went wrong.

Detecting drift without labeling every live query. The practical answer isn't waiting for enough labeled production data to retrain a judge — it's running a small, fixed evaluation set (the same kind of curated question/reference set already used throughout this site, via RAGAS or the metrics on the Evaluation Metrics page) against the live system on a recurring schedule, not just at deploy time, and alerting when the score on that fixed set drops. The alert fires because measured quality dropped, not because anything threw an exception — which is exactly the class of failure a normal error-rate dashboard is structurally blind to. A cheaper complementary practice: a periodic manual spot-check of a random sample of real production answers against source documents, to catch failure patterns the fixed eval set doesn't happen to cover.

Scalability: real numbers, not estimates

Embedding throughput is a GPU-allocation and region-placement problem at real scale, not a code-optimization one. A real, published example: generating embeddings for ~30 million product reviews with a 7B-parameter embedding model on a single-region setup needed 47 L4 GPUs and took 20 hours at $710; spreading the same job across 12 regions (mixing spot and on-demand instances to find spare capacity in less-contended regions) cut that to 2.3 hours at $277 — a 9× speedup and 61% cost reduction from placement alone, no model or code change. Batch-size and precision choices matter too: length-based batch sorting to minimize padding waste is commonly cited as a 20–40% compute saving, and mixed-precision (FP16) inference cuts memory with minimal quality loss. On the buy-vs-build question specifically, a real cost-crossover exists: API-based embedding pricing (roughly $0.02–0.06 per million tokens from major providers) beats self-hosting a dedicated GPU below roughly 50–100 million tokens per month of embedding volume — self-hosting only wins once throughput is high enough to keep that GPU consistently busy, not from the first token processed.

Right-sizing every auxiliary LLM call is a scalability lever, not just a cost one. Every pipeline-stage LLM call beyond the main generation — routing, relevance grading (CRAG's post-retrieval evaluator), reflection/critique (Self-RAG's [IsRel]/[IsSup] tokens), reranking, compression — is structurally a narrow classification or scoring task, not open-ended generation, and doesn't need a frontier-model-sized call to do it well. Recent research makes this argument directly for RAG specifically: using full-scale LLMs for binary routing/grading decisions in a high-throughput pipeline introduces real, avoidable computational redundancy, and a small, purpose-tuned model can match a much larger general model's routing accuracy at roughly an order of magnitude lower latency. This is why small/cheap models for grading and routing shows up repeatedly across Self-RAG, CRAG, and Agentic RAG — it's the same right-sizing principle applied at every stage that isn't the final answer generation itself, and it compounds: a system with three auxiliary LLM calls per query pays that "narrow task on a frontier model" tax three times over if none of them are right-sized.

This page's own pipeline, measured

def latency_breakdown(question: str, chunks: list, embed_fn, generate_fn) -> dict:
    """Real per-stage timing across a naive RAG pipeline, not just an
    end-to-end number -- `embed_fn(text)` returns a normalized
    embedding, `generate_fn(prompt)` returns a generated string."""
    import time

    import numpy as np

    timings = {}

    start = time.monotonic()
    query_vec = embed_fn(question)
    timings["embed_query"] = time.monotonic() - start

    start = time.monotonic()
    chunk_embeddings = np.array([embed_fn(c) for c in chunks])
    timings["embed_corpus"] = time.monotonic() - start

    start = time.monotonic()
    sims = chunk_embeddings @ query_vec
    top_k = np.argsort(-sims)[:3]
    retrieved = [chunks[i] for i in top_k]
    timings["retrieval_search"] = time.monotonic() - start

    start = time.monotonic()
    context_block = "\n---\n".join(retrieved)
    generate_fn(f"Answer the question using the context below.\n\nContext:\n{context_block}\n\nQuestion: {question}")
    timings["generation"] = time.monotonic() - start

    return timings

def track_generation_usage(prompt: str) -> dict:
    """Extracts real per-call token usage from a local Ollama response
    -- `prompt_eval_count` and `eval_count` are the same shape of data
    a hosted provider exposes via `response.usage`, so this aggregation
    pattern is provider-agnostic."""
    import requests

    response = requests.post(
        "http://localhost:11434/api/generate",
        json={"model": "qwen3:4b", "prompt": prompt, "stream": False},
        timeout=180,
    ).json()
    return {"prompt_tokens": response.get("prompt_eval_count", 0), "completion_tokens": response.get("eval_count", 0)}

Real per-stage latency for a single query against this page's naive RAG pipeline:

embed_query        0.561s  (  5.4%)
embed_corpus       0.457s  (  4.4%)
retrieval_search   0.026s  (  0.2%)
generation         9.343s  ( 90.0%)
total              10.387s

Generation dominates end-to-end latency by a wide margin — the same pattern that makes the small-model-for-auxiliary-calls argument above concrete: shaving milliseconds off retrieval matters far less here than making sure every non-essential LLM call in a pipeline isn't accidentally running on the expensive model.

Real per-call token usage, aggregated across 4 questions against a local Ollama model:

generation: 4 calls, 1333 prompt tokens, 2608 completion tokens
avg prompt tokens/call: 333
avg completion tokens/call: 652

Completion tokens nearly double prompt tokens here — a real, measured consequence of using a reasoning-capable local model that generates visible thinking tokens before its final answer, worth knowing since it's a genuine, provider-specific cost driver that a naive "prompt tokens are usually the bulk of the cost" assumption would miss.

Scenario Check

Sources & further reading