Skip to content

Modular RAG

Every technique on the pre-retrieval and post-retrieval pages is an individual fix bolted onto a fixed retrieve → generate pipeline. Modular RAG is a step up in abstraction: treat indexing, pre-retrieval, retrieval, post-retrieval, and generation as independent, swappable modules that different queries can move through differently, rather than one path everyone takes. Adaptive RAG, Self-RAG, Corrective RAG, and Agentic RAG are all specific instances of this idea — each one is "modular RAG plus a particular way of deciding which module runs."

Every one of those routers, though, makes its decision with an LLM call. This page asks the question none of them tested: does routing actually need one?

Hands-on

The full 3-way comparison below is runnable, against real data: Modular RAG → in the code repo. Two of the three routers need no LLM key at all.

TL;DR — quick revision
  • The same routing decision, built 3 ways: keyword heuristics (no model call), embedding-based semantic routing (one local embedding call, no LLM), and the exact LLM router already built in adaptive-rag/
  • Measured on two separate sets, deliberately: a "design set" (the queries the heuristic rules were written looking at) and a genuinely held-out set (written without looking at those rules) — because a router's accuracy on the queries it was tuned against tells you almost nothing about how it'll do on the next query someone actually asks
  • The heuristic router's accuracy collapses from 1.00 to 0.40 moving from the design set to held-out phrasing — real, measured brittleness, not a hypothetical caveat about rule-based systems
  • The embedding-based router is the more honest number: 0.67 → 0.80 across the same two sets — roughly consistent, because it was never fit to either one
  • The LLM router is the only one both accurate and consistent everywhere — 1.00 on both sets — at roughly 30-1500x the latency of the other two, and a real per-call cost they don't have

What "modular" actually means here

A modular RAG system separates the pipeline into pieces that can each be swapped independently:

  • Indexing — chunking/embedding strategy, potentially different per document type
  • Pre-retrieval — rewriting, HyDE, decomposition, step-back — chosen per query, not applied blindly to every query (the entire lesson of the pre-retrieval page)
  • Routing — deciding which retrieval path or index to use at all
  • Retrieval — sparse, dense, or hybrid, against whichever index routing picked
  • Post-retrieval — reranking, compression, applied conditionally
  • Generation — a cheap model for simple lookups, an expensive one for reasoning-heavy questions

The routing module is the piece this page focuses on, because it's the one every other advanced technique on this site quietly assumes works, without ever measuring the cheaper alternatives to the LLM call each one reaches for by default.

The same decision, three ways

All three routers solve the identical problem: classify a query into one of 5 categories (CLEAR, VAGUE, COMPOUND, BROAD, NEEDS_CONTEXT) — the same categories Adaptive RAG uses to pick a pre-retrieval technique.

Heuristic routing — pattern-matching, zero cost

The cheapest possible option: a handful of keyword and pattern rules, evaluated in plain Python with no model call at all. Fast by construction — there's no inference step to be slow.

COMPOUND_MARKERS = [" and ", " as well as "]
VAGUE_MARKERS = ["dont", "didnt", "just ", "how come", "everyone", "like everyone"]
BROAD_MARKERS = ["why is", "why did", "what lets", "what makes", "how does"]
NEEDS_CONTEXT_MARKERS = ["value", "rate did", "parameter"]


def heuristic_router(query: str) -> str:
    q = query.lower()

    if any(marker in q for marker in COMPOUND_MARKERS) and q.count("?") <= 1 and "," in query:
        return "COMPOUND"
    if any(marker in q for marker in VAGUE_MARKERS):
        return "VAGUE"
    if any(marker in q for marker in NEEDS_CONTEXT_MARKERS):
        return "NEEDS_CONTEXT"
    if any(q.startswith(marker) or marker in q for marker in BROAD_MARKERS):
        return "BROAD"
    return "CLEAR"

Embedding-based semantic routing — one local embedding call, no LLM

A middle ground: write a few example utterances per category once, embed them (offline, one-time cost), and average them into a centroid vector per category. At query time, embed the incoming query and route to whichever centroid it's closest to by cosine similarity — no LLM call anywhere in this path, just the same embedding model already used for retrieval.

import numpy as np
from sentence_transformers import SentenceTransformer

EXAMPLE_UTTERANCES = {
    "CLEAR": [
        "What dataset was used for evaluation?",
        "What programming language was used?",
        "Which company published this paper?",
    ],
    "VAGUE": [
        "why didnt they just do it the old way",
        "how come nobody thought of this before",
        "whats the deal with skipping that step",
    ],
    "COMPOUND": [
        "What is the accuracy, and what is the training time?",
        "How many layers are there, and what activation function is used?",
        "What is the cost, and what is the latency?",
    ],
    "BROAD": [
        "Why does this approach outperform previous methods?",
        "What makes this design more efficient overall?",
        "How does this system achieve better generalization?",
    ],
    "NEEDS_CONTEXT": [
        "What value was used for the learning rate?",
        "What rate did they set for weight decay?",
        "What parameter controls the batch size?",
    ],
}


def build_router_index(embed_model: SentenceTransformer) -> dict[str, np.ndarray]:
    """One-time setup: embed each category's example utterances and
    average them into a single centroid vector per category."""
    centroids = {}
    for category, examples in EXAMPLE_UTTERANCES.items():
        embeddings = embed_model.encode(examples, convert_to_numpy=True, show_progress_bar=False)
        centroids[category] = embeddings.mean(axis=0)
    return centroids


def embedding_router(query: str, centroids: dict[str, np.ndarray], embed_model: SentenceTransformer) -> str:
    query_vec = embed_model.encode(query, convert_to_numpy=True)
    sims = {
        category: float(np.dot(query_vec, centroid) / (np.linalg.norm(query_vec) * np.linalg.norm(centroid) + 1e-8))
        for category, centroid in centroids.items()
    }
    return max(sims, key=sims.get)

LLM-based routing — the router already built in adaptive-rag/

The most expensive, most flexible option, reused directly rather than rebuilt: the exact same prompt and parsing logic from adaptive-rag/, included here specifically as the accuracy baseline the other two get measured against.

import re

import llm

CATEGORIES = ["CLEAR", "VAGUE", "COMPOUND", "BROAD", "NEEDS_CONTEXT"]

ROUTER_PROMPT = """Classify the query below into EXACTLY ONE category, \
based on what problem (if any) it poses for retrieval against a \
technical document. Categories, each with an example:

CLEAR - a single specific fact that is a NAMED thing (an algorithm \
name, a hardware type, a dataset name) — not a numeric parameter value \
— already phrased using the kind of technical vocabulary the document \
itself would use.
  Example: "What optimizer did they use for training?"

VAGUE - a single specific fact whose complete answer is ONE short \
thing, but the question is phrased colloquially, avoiding the \
technical vocabulary the document actually uses.
  Example: "why dont they just use the old-school approach everyone \
used before"

COMPOUND - explicitly asks for two or more distinct, independently \
answerable facts, usually joined by "and".
  Example: "What optimizer did they use, and how many layers does it have?"

BROAD - one open-ended question whose COMPLETE answer requires citing \
several distinct reasons or mechanisms (not one short fact) — decide \
based on whether a full answer needs multiple points, regardless of \
whether the wording sounds casual or technical.
  Example: "What lets this model handle relationships between things \
that are positioned far apart, without losing track over long spans?"

NEEDS_CONTEXT - asks for one narrow NUMERIC or configuration parameter \
value (a rate, a dimension, a count) where knowing the broader \
principle/section it belongs to would help interpret or locate the \
answer.
  Example: "What value did they use for the regularization strength?"

Query: {query}

Respond with ONLY the category name, wrapped in <category></category> tags."""


def llm_router(question: str, provider: str | None = None) -> str:
    response = llm.generate(ROUTER_PROMPT.format(query=question), provider=provider)
    match = re.search(r"<category>(.*?)</category>", response, re.S)
    category = match.group(1).strip() if match else "CLEAR"
    return category if category in CATEGORIES else "CLEAR"

Measuring all three honestly — and why two test sets, not one

Testing a router only on the queries used to design it answers the wrong question. A heuristic router's keyword rules are, almost by definition, going to match the exact phrasings someone had in mind while writing them — a 100% score there proves the rules were written correctly, not that they generalize. So this comparison uses two separate sets: a 9-query "design set" (the same queries adaptive-rag/'s router was originally validated on) and a 5-query held-out set, written without looking at the heuristic router's actual keyword list — the fair test of whether a router generalizes, not just fits.

Router Design-set accuracy Held-out accuracy Avg latency
Heuristic 1.00 0.40 ~0ms
Embedding 0.67 0.80 ~50ms
LLM 1.00 1.00 ~1500ms

The headline number to distrust is the heuristic's 1.00

On the design set, the heuristic router gets every single query right — a keyword like "dont" or "and" matching exactly the patterns it was built around. Tested on the held-out set — "What is the batch size, plus what hardware did they train on?", "What accounts for this model generalizing so well to new domains?", "What dropout probability was applied?" — accuracy collapses to 2 out of 5. Real questions people actually ask don't reliably contain the exact keywords a rule-writer anticipated; this is measured brittleness, not a hypothetical caveat about heuristics in general.

The embedding router's number is the more trustworthy one

0.67 on the design set, 0.80 on held-out — genuinely similar, because the embedding router was never fit to either set; its example utterances were written independently of both. Its actual confusion pattern is consistent and diagnosable too: it repeatedly misroutes distinctly-phrased COMPOUND questions to NEEDS_CONTEXT — a real, specific weakness in how compound-question phrasing sits in embedding space relative to the example utterances chosen for NEEDS_CONTEXT, not a mysterious black box.

The LLM router costs what you'd expect for being right everywhere

~1500ms average per classification call, against ~0ms and ~50ms for the other two, plus a real per-call API cost neither of the others has at all. It earns that cost here by being the only router that's both accurate and consistent across both test sets — but that's specifically the trade-off worth measuring before defaulting to an LLM call for every routing decision in a pipeline, the same way every earlier page on this site has argued against applying any technique unconditionally.

Concrete example — a customer support system

A modular pipeline doesn't have to use the same router (or even the same routing mechanism) for every decision it needs to make:

  • "What's your refund policy?" → cheap, high-confidence category (a simple FAQ lookup) → heuristic or embedding routing is fine here, since the cost of an occasional misroute is low and the volume is high
  • "Compare enterprise vs. individual plan refund timelines across regions" → a genuinely ambiguous, high-stakes routing decision (get this wrong and the whole downstream pipeline — decomposition, hybrid retrieval, reranking — runs on the wrong plan) → the LLM router's cost is worth it here

Same system, two different routing mechanisms, chosen per decision based on how much a misroute actually costs — the same idea as choosing generation model size per query, applied one level upstream.

Connection to everything else on this site

Self-RAG and Corrective RAG are modular RAG systems where the orchestration module includes a self-evaluation/correction loop. Agentic RAG goes one step further — an LLM is the orchestration module, making the routing/looping decision itself at runtime rather than following a rule someone wrote in advance, whether that rule is a keyword, a cosine-similarity threshold, or a fixed if/else branch. All three are the same underlying idea (modules, chosen dynamically) with a different mechanism deciding which module runs.

Use it for / skip it for

Use LLM-based routing for: decisions where being wrong is expensive and query phrasing is genuinely unpredictable — the accuracy/consistency this page measured is worth the latency there. Use heuristic or embedding-based routing for: high-volume, well-defined categories where the cost of an occasional misroute is low, and where you can actually validate on a real held-out set before trusting the numbers — not just the queries used to build the rules.

Scenario Check

Sources & further reading