Skip to content

Agentic RAG

Every technique before this one on the site still has a human — or at least a pre-written prompt — deciding the control flow. Adaptive RAG's router picks a pre-retrieval technique via a classification prompt. Corrective RAG and Self-RAG check something and branch, but the branches themselves ("if Incorrect, web search") are rules someone wrote in advance. Graph RAG even ended with an explicit gap: choosing local vs. global search was a manual flag, not something the system decided. Agentic RAG is what closes that gap — instead of code deciding what happens next, an LLM agent decides for itself, at runtime, which tool to reach for.

Hands-on

The full agent loop below is runnable, against real data: Agentic RAG → in the code repo. This recipe's agent loop uses Anthropic's tool-calling API directly, so it needs an Anthropic key specifically (not the pluggable provider setup used elsewhere in this repo) — see below for why.

TL;DR — quick revision
  • The core mechanism is a ReAct loop (Reason, Act, Observe, repeat) — but built here with the model's actual tool-calling API, not prompted text parsed with regex or tags the way every earlier recipe's routing worked
  • Four tools, each a real technique already verified elsewhere on this site: plain vector search, graph local search, graph global search, and a real web search fallback — the agent picks among them itself
  • Verified correct on all 4 distinct question types, unprompted: a simple factual question, a broad synthesis question, a specific relational question, and a question entirely outside the corpus — no routing logic was written for any of it; the tool descriptions alone were enough
  • This is genuinely more expensive and less predictable than everything before it — the number of tool calls varies per question, and a misinterpreted result can send the loop in an unproductive direction for several steps, which is why every real implementation needs a hard iteration cap and an honest "ran out of budget" exit
  • Not free lunch, and not for everything — high-volume, well-understood query patterns are cheaper and more predictable with a fixed or modular pipeline; agentic control flow earns its cost specifically when the right retrieval strategy genuinely can't be known in advance

The ReAct loop, with real tool calls

Every prior "routing" technique on this site worked by asking an LLM to output a classification, then code parsed that text (a <category> tag, a comma-separated list) and branched on it by hand. That's a real, working pattern — but it's still code deciding what to do with the model's answer. Tool calling flips that: the model itself emits a structured tool_use request — a specific tool name and validated arguments — and the only thing left for code to do is actually run that tool and hand the result back. The model decides which tool, with what input, and whether it needs another one after seeing the result.

Here are the four tools this agent gets to choose from, each one a technique already built and measured on its own page elsewhere on this site:

TOOLS = [
    {
        "name": "naive_vector_search",
        "description": (
            "Plain vector similarity search over the document. Best for a single, "
            "specific factual question already phrased using the document's own vocabulary."
        ),
        "input_schema": {
            "type": "object",
            "properties": {"query": {"type": "string", "description": "The search query."}},
            "required": ["query"],
        },
    },
    {
        "name": "graph_local_search",
        "description": (
            "Searches a knowledge graph by linking the query to specific entities and "
            "traversing their relationships. Best for questions about how two or more "
            "specific things are connected to each other."
        ),
        "input_schema": {
            "type": "object",
            "properties": {"query": {"type": "string", "description": "The search query."}},
            "required": ["query"],
        },
    },
    {
        "name": "graph_global_search",
        "description": (
            "Searches using summaries of the whole knowledge graph's community structure. "
            "Best for broad questions that require synthesizing across many different parts "
            "of the document — e.g. 'what are the main components/themes' — where no single "
            "entity or chunk has the complete answer."
        ),
        "input_schema": {
            "type": "object",
            "properties": {"query": {"type": "string", "description": "The search query."}},
            "required": ["query"],
        },
    },
    {
        "name": "web_search",
        "description": (
            "Searches the public web. Use this when the question is about something outside "
            "this technical paper entirely (current events, unrelated general knowledge), or "
            "when the other tools returned nothing useful for a question that clearly needs "
            "external information."
        ),
        "input_schema": {
            "type": "object",
            "properties": {"query": {"type": "string", "description": "The search query."}},
            "required": ["query"],
        },
    },
]

Notice what's actually doing the routing work here: not a classifier prompt, not a set of rules — just the description field on each tool. The model reads these the same way it would read documentation for any function it's deciding whether to call.

What each tool actually does when called isn't repeated here — every one of them is real, working code already shown and explained in full on its own page, and this page would just be duplicating it:

  • naive_vector_search — plain bi-encoder top-k, the same retrieval step walked through on Naive RAG
  • graph_local_search / graph_global_search — entity-linking + traversal, and map-reduce over community summaries, both fully explained on Graph RAG
  • web_search — a real Tavily call, the same fallback mechanism used on Corrective RAG

If you want to see the actual implementations side by side with this page's agent loop, agentic_rag.py in the repo has all four wired together in one file — worth reading directly rather than inferring from the tool descriptions alone, especially before adapting this pattern for your own tools.

The loop itself

With those four tools declared, the agent loop is genuinely simple: ask the model, check whether it wants to call a tool, run the tool if so, feed the result back, and repeat — stopping either when the model has enough to answer directly, or when a hard cap is hit.

import os

import anthropic

MAX_AGENT_ITERATIONS = 6


def agentic_rag(question: str, dispatch_tool, tools: list[dict] = TOOLS, max_iterations: int = MAX_AGENT_ITERATIONS) -> dict:
    """A ReAct-style loop, made of real API tool_use blocks: Claude
    picks a tool (Action), the tool runs and its output is fed back
    (Observation), and Claude decides — for itself, each iteration —
    whether it has enough to answer or needs to call another tool
    (Thought). Capped at max_iterations to avoid an unproductive loop
    running forever; if the cap is hit, an honest partial answer beats
    silently hanging.

    `dispatch_tool(name, tool_input) -> str` executes whichever tool
    Claude picked and returns its result as text — see
    agentic_rag.py's `_dispatch_tool` for the real implementation that
    routes to naive_vector_search / graph_local_search /
    graph_global_search / web_search."""
    client = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
    model = os.environ.get("ANTHROPIC_MODEL", "claude-sonnet-4-5")
    messages = [{"role": "user", "content": question}]
    trace = []

    for _ in range(max_iterations):
        response = client.messages.create(model=model, max_tokens=2048, tools=tools, messages=messages)

        if response.stop_reason != "tool_use":
            final_text = next((b.text for b in response.content if b.type == "text"), "")
            return {"trace": trace, "answer": final_text, "status": "ok"}

        messages.append({"role": "assistant", "content": response.content})

        tool_results = []
        for block in response.content:
            if block.type == "tool_use":
                result = dispatch_tool(block.name, block.input)
                trace.append({"tool": block.name, "input": block.input, "result_preview": result[:200]})
                tool_results.append({"type": "tool_result", "tool_use_id": block.id, "content": result})

        messages.append({"role": "user", "content": tool_results})

    return {"trace": trace, "answer": "Reached the iteration cap without a final answer.", "status": "max_iterations"}

The iteration cap matters for a reason that has nothing to do with cost: a genuinely open-ended loop, given a misleading tool result, has no built-in reason to ever stop trying. Capping it and returning an honest "reached the iteration limit without a final answer" is the same discipline as Self-RAG's regenerate-once-then-fall-back-honestly pattern, applied to a loop whose length isn't fixed in advance.

Verified: four different questions, four different (correct) tool choices

No routing logic was written anywhere in this recipe — the agent picked every one of these itself, from the tool descriptions alone.

A simple factual question → naive_vector_search

Asked "What optimizer did they use for training?" — the agent called naive_vector_search({'query': 'optimizer training'}) on the first step, got back the Adam-optimizer passage directly, and answered from it without needing anything else. The simplest tool for the simplest question — correctly recognized as such.

A broad synthesis question → graph_global_search

Asked "What are the main components of the Transformer architecture and how do they work together?" — the same broad question Graph RAG measured naive top-5 retrieval failing on — the agent correctly reached for graph_global_search, and produced a genuinely comprehensive answer covering the encoder, decoder, multi-head attention mechanics, positional encoding, feed-forward networks, and residual connections. The tool's description ("best for broad questions that require synthesizing across many different parts") was enough to steer it here over the cheaper, narrower options.

A specific relational question → graph_local_search

Asked "What connects label smoothing to the BLEU score improvements?" — the same question Graph RAG's local search demonstrated a real win on — the agent called graph_local_search, matched the entities Label Smoothing and BLEU Score directly, and answered from the traversed connection. It didn't reach for the more expensive global search or a plain vector lookup; the specific, two-entity nature of the question matched the tool built for exactly that.

A question outside the corpus entirely → web_search

Asked "What is the current stock price of Nvidia?" — nothing in a 2017 machine learning paper could possibly answer this — the agent correctly skipped straight to web_search, made a real Tavily call, and returned genuinely current stock data (with the honest caveat that different sources in the results disagreed on the exact price, since stock prices move by the second). It didn't waste a call on naive_vector_search first and fail — it recognized the question was out of scope for the corpus-specific tools from the description alone.

Why this needs more than everything before it

Tool diversity beyond retrieval. Nothing here restricts the agent to only choosing between retrieval strategies — the same mechanism extends to a calculator, a SQL database, a code execution tool, or another specialized agent, combined in whatever order a question actually demands. A fixed pipeline has to guess the tool sequence upfront; an agent works it out as it goes, including cases needing more than one tool in sequence.

Real costs that don't show up on a fixed pipeline. Latency and cost per query are no longer predictable the way a fixed 3-stage pipeline's are — a simple question might resolve in one tool call, a harder one in four or five. Error compounding is real: a misread tool result can send the loop down an unproductive path, which is exactly why the iteration cap above isn't optional. And evaluation gets harder — a fixed pipeline can be graded stage by stage (retrieval metrics, then generation metrics); an agent's path itself varies per question, so evaluating it well means reviewing whether each tool call was reasonable, not just whether the final answer was right.

Framing within LangGraph / CrewAI / AutoGen

LangGraph defines an explicit graph of nodes (retrieve, grade, generate, call a tool) and edges — including conditional and looping edges — giving agentic flexibility while still constraining which paths are even possible, a middle ground between a free-form agent and a fully fixed pipeline. CrewAI / AutoGen lean toward multi-agent setups — several specialized agents (a retriever agent, a critic agent, a writer agent) collaborating or handing off tasks, useful specifically when one agent's context or role would get overloaded trying to do everything itself.

Use it for / skip it for

Use it for: genuinely open-ended, multi-hop, multi-source questions where the retrieval strategy can't be enumerated in advance — research assistants, complex analyst tools, multi-system troubleshooting. Skip it for: high-volume, well-understood query patterns — simple FAQ lookups, single-document Q&A — where a fixed or modular pipeline is faster, cheaper, and just as accurate; agentic overhead buys nothing when the right strategy was already knowable in advance.

Scenario Check

Sources & further reading