Skip to content

RL for Search and Tool Agents

Hands-on

Full runnable recipe: rl-search-tool-agents/ in the companion cookbook — a real repro of ToolRL's reward-design claim on real tool-call completions. Actually training these methods needs GPU-hours this cookbook doesn't have, so — per this site's own standing decision for training content — the rest stays conceptual, grounded in each paper's own verified abstract and results.

TL;DR — quick revision
  • Search-R1 trains an LLM to interleave real search calls with reasoning, using outcome-only reward. Real, verified: "Search-R1 optimizes LLM reasoning trajectories with multi-turn search interactions... Experiments on seven question-answering datasets show that Search-R1 improves performance by 41% (Qwen2.5-7B) and 20% (Qwen2.5-3B) over various RAG baselines." R1-Searcher and ReSearch train the same joint reasoning+search behavior via RL with no supervised reasoning-step data at all — R1-Searcher via a two-stage outcome-only process, ReSearch by treating search as guided by the model's own "text-based thinking."
  • ToolRL studied reward design for tool use directly, and found coarse reward isn't enough. Real, verified: "coarse-grained reward signals, such as answer matching, fail to offer the finegrained feedback required for effective learning." Its fix — decomposing reward into tool-name, parameter-name, and parameter-value components — produced "a 17% improvement over base models and a 15% gain over SFT models." This page's own repro reproduces the exact mechanism on real tool calls.
  • ReTool shows RL can teach WHEN to reach for a tool, not just how. Real, verified: on AIME, "67% accuracy with 400 training steps" vs. a text-only RL baseline's "40% accuracy, 1080 steps" — higher accuracy in roughly a third of the training steps — with the paper describing an emergent "aha moment" where the model starts self-correcting code mid-reasoning without being explicitly taught to.
  • RAGEN names a real, recurring multi-turn RL failure mode: the Echo Trap. Real, verified: "a recurring instability pattern, Echo Trap, where agents overfit to locally rewarded reasoning patterns, marked by reward variability collapse, entropy drop, and gradient spikes." Models converge to "near-identical phrasing... without justification" — real, multi-turn RL training doesn't just get slower when it goes wrong, it can collapse into repeating a shallow, memorized pattern that happens to score well locally.

Three real, separate 2026 papers converge on the same shape: train an LLM to decide for itself, during reasoning, when to issue a search query — using only whether the final answer was right, no supervised data on the reasoning steps themselves.

Search-R1 is explicit about the mechanism: the model "learns to autonomously generate (multiple) search queries during step-by-step reasoning with real-time retrieval," using dedicated tokens to mark search calls and the retrieved results returned into the same rollout, trained with either PPO or GRPO and "a simple outcome-based reward function." The real, verified headline: "Search-R1 improves performance by 41% (Qwen2.5-7B) and 20% (Qwen2.5-3B) over various RAG baselines under the same setting" — across seven real QA datasets.

R1-Searcher takes a two-stage route to the same behavior: "a novel two-stage outcome-based RL approach designed to enhance the search capabilities of LLMs," which "relies exclusively on RL, without requiring process rewards or distillation for a cold start" — stage one teaches the model to invoke search at all, stage two teaches it to use search effectively toward a correct answer. Its real, verified claim: the method "significantly outperforms previous strong RAG methods, even when compared to the closed-source GPT-4o-mini."

ReSearch frames the same joint training slightly differently: search isn't a separately triggered action bolted onto reasoning, it's "guided by text-based thinking" as an integral part of the reasoning chain itself, with "search results subsequently influenc[ing] further reasoning." Real and notable: the paper reports the trained models "naturally elicit advanced reasoning capabilities such as reflection and self-correction" as a side effect of the RL process — nobody wrote a reflection-training objective; it emerged from training the model to get multi-hop answers right via search.

The common thread across all three: none of them supervise how to search. They only ever reward whether the final answer was correct, and let RL discover the search strategy — which is exactly why reward design for the intermediate tool calls themselves becomes its own real, separate research question.

ToolRL: reward design isn't an afterthought, it's the actual lever

ToolRL asks a narrower, sharper question: given that a tool call itself (not just a final text answer) is what's being scored, what should the reward function for that actually look like? Its own framing of the gap: "multiple tools may be invoked with diverse parameters, and coarse-grained reward signals, such as answer matching, fail to offer the finegrained feedback required for effective learning." The paper's fix, verified: decompose the reward into per-component scores — tool selection, parameter names, parameter values — rather than one all-or-nothing match. Trained with GRPO, the real, verified result: "a 17% improvement over base models and a 15% gain over SFT models."

Repro: what binary reward can't see, on real tool calls

The code below implements both a binary (exact-match) reward function and ToolRL's decomposed, component-by-component version, then runs both against a set of real tool-call completions to see where the two scoring methods actually disagree.

def binary_reward(call: dict, expected: dict) -> float:
    """The coarse, 'answer matching' reward ToolRL's abstract names as the failure case."""
    return 1.0 if call == expected else 0.0


def fine_grained_reward(call: dict, expected: dict) -> dict:
    """ToolRL's own three-component decomposition: tool name, parameter names,
    parameter values -- each scored separately, then averaged."""
    tool_name_score = 1.0 if call.get("tool") == expected.get("tool") else 0.0

    expected_params = expected.get("params", {})
    call_params = call.get("params", {})
    expected_keys, call_keys = set(expected_params), set(call_params)
    param_name_score = (
        len(expected_keys & call_keys) / len(expected_keys) if expected_keys else 1.0
    )

    if expected_keys:
        value_matches = sum(1 for k in expected_keys if call_params.get(k) == expected_params.get(k))
        param_value_score = value_matches / len(expected_keys)
    else:
        param_value_score = 1.0

    total = round((tool_name_score + param_name_score + param_value_score) / 3, 4)
    return {
        "tool_name_score": tool_name_score,
        "param_name_score": round(param_name_score, 4),
        "param_value_score": round(param_value_score, 4),
        "total": total,
    }

_TRIALS = [
    {
        "label": "fully_specified",
        "prompt": "Create a calendar event: title 'Team Sync', date 2026-10-05, duration 30 minutes.",
        "expected": {"tool": "create_calendar_event", "params": {"title": "Team Sync", "date": "2026-10-05", "duration_minutes": 30}},
    },
    {
        "label": "vague_duration",
        "prompt": "Create a calendar event titled 'Team Sync' on 2026-10-05. Keep it brief, like a quick check-in.",
        "expected": {"tool": "create_calendar_event", "params": {"title": "Team Sync", "date": "2026-10-05", "duration_minutes": 30}},
    },
    {
        "label": "decoy_tool_available",
        "prompt": "Remind me about Team Sync on 2026-10-05.",
        "expected": {"tool": "create_calendar_event", "params": {"title": "Team Sync", "date": "2026-10-05", "duration_minutes": 30}},
    },
    {
        "label": "paraphrased_duration",
        "prompt": "Set up a 'Team Sync' meeting for 2026-10-05, half an hour long.",
        "expected": {"tool": "create_calendar_event", "params": {"title": "Team Sync", "date": "2026-10-05", "duration_minutes": 30}},
    },
]

Four real trials against the identical target action — a 30-minute "Team Sync" event on 2026-10-05 — each phrased to elicit a different, real kind of imperfection: fully specified, a vaguely-worded duration, a request that could plausibly map to a different available tool, and a paraphrased duration.

A real run, unforced — binary vs. fine-grained reward on the same real completions

Trial Real call Binary reward Fine-grained reward
Fully specified Exact match 1.0 1.0
Vague duration ("keep it brief") Right tool, right params, guessed duration_minutes: 15 vs. expected 30 0.0 0.8889
Decoy tool available Called create_reminder instead of create_calendar_event 0.0 0.2222
Paraphrased duration ("half an hour") Exact match 1.0 1.0

The real finding is the middle two rows: binary reward scores the "vague duration" call and the "decoy tool" call identically — both 0.0 — even though one call got the tool right and every parameter right except a single reasonably-guessed value, and the other picked an entirely different tool. A policy trained on binary reward alone gets zero gradient signal telling those two real, different completions apart. Fine-grained reward gives them real, different scores — 0.8889 vs. 0.2222 — the exact differentiation ToolRL's own abstract argues coarse reward can't provide.

ReTool: RL can teach when, not just how

ReTool targets a different question: given a model that already knows how to write and run code, when during a reasoning chain should it actually reach for that tool? Its approach combines a cold-start phase — "synthetic cold-start data generation to produce code-augmented long-form reasoning traces" — with RL that "leverages task outcomes as rewards to iteratively refine the model's tool use strategy, enabling autonomous discovery of optimal tool invocation patterns." The real, verified numbers, on AIME with a 32B model: 67% accuracy with 400 training steps, against a text-only RL baseline's 40% accuracy at 1080 steps — meaningfully higher accuracy in roughly a third of the training steps. The paper also documents a real, named emergent behavior: "code self-correction, signaling an 'aha moment' in which the model autonomously masters adaptive tool use" — the model wasn't explicitly trained to notice and fix its own buggy code mid-reasoning; that behavior showed up as a byproduct of outcome-based RL over enough training steps.

RAGEN: naming what goes wrong in multi-turn agent RL

Every method above assumes multi-turn RL training actually converges cleanly. RAGEN studies what happens when it doesn't, across four stylized multi-turn agent environments, and names a real, recurring failure mode directly: "a recurring instability pattern, Echo Trap, where agents overfit to locally rewarded reasoning patterns, marked by reward variability collapse, entropy drop, and gradient spikes." Concretely, the paper reports models converging "to near-identical phrasing focused on choosing [an action] without justification" — the policy doesn't fail by getting worse in an obvious way; it collapses into a narrow, repetitive, shallow pattern that happens to score adequately under the local reward signal, at the cost of the actual reasoning the training was meant to produce.

RAGEN's proposed fix, StarPO-S, addresses this with "trajectory filtering, critic incorporation, and gradient stabilization." The paper's third real finding sharpens exactly why ToolRL's reward-design question and RAGEN's stability question are connected: "without fine-grained, reasoning-aware reward signals, agent reasoning hardly emerge[s] through multi-turn RL and they may show shallow strategies or hallucinated thoughts." Coarse reward doesn't just fail to differentiate good completions from bad ones (ToolRL's finding) — across many real turns of RL training, it can actively drive the policy toward the Echo Trap's collapsed, shallow behavior, because a shallow, repeated pattern is often the path of least resistance to a locally-adequate reward.

What this means in practice

The six real methods on this page split cleanly into two real, connected layers. Search-R1, R1-Searcher, and ReSearch answer "should the model decide when to call a tool during reasoning?" — yes, and outcome-only RL is enough to teach that, without any supervised reasoning-step data. ToolRL and ReTool answer "once it's decided to call a tool, how do you reward that specific action well?" — coarse, all-or-nothing reward isn't enough; decomposed, fine-grained reward (ToolRL) and outcome-based iterative refinement over many real training steps (ReTool) both do measurably better. RAGEN answers the question underneath both: "what happens when the reward signal training either of these is too coarse, over many multi-turn rollouts?" — real, documented instability, named and diagnosable, not just "training got noisy."

Interview angle

Weak answer to "how would you train an agent to use tools better with RL?": "Give it a reward of 1 for a correct final answer and 0 otherwise, and run PPO." This page's own real repro is a direct, measured counterexample at the level of a single tool call: binary reward assigned the identical score (0.0) to a call that was almost entirely correct and a call that used the wrong tool entirely — real, zero differentiation between two very different real mistakes.

Strong answer: name the two real, separate design questions this page distinguishes — whether to let the model decide when to act (Search-R1/R1-Searcher/ReSearch show outcome-only RL is sufficient for that) versus how to reward the specific action once taken (ToolRL and ReTool show this needs finer-grained signal, and ReTool shows RL can improve timing specifically, not just correctness). Then name the real risk of getting the reward too coarse across many turns: RAGEN's Echo Trap, a real, documented case where multi-turn RL collapses into a shallow, locally-rewarded pattern rather than the reasoning it was meant to produce.

Follow-up to expect: "how would you detect an Echo Trap in your own training run before it wastes a lot of compute?" RAGEN's own real, verified signature gives a direct, checkable answer: watch for reward variability collapsing, entropy dropping, and gradient spikes appearing together — not just a flattening loss curve, which could mean convergence rather than collapse. The qualitative tell this page's own material adds: sample actual rollouts periodically and check whether the model's stated reasoning is getting more repetitive and less justified over training, since RAGEN's own finding is that the model can keep scoring adequately while doing exactly that.

Build it yourself — 30 minutes

  1. Pick a real tool-calling task with more than one parameter, and write both a binary reward function (exact match) and a decomposed reward function (component-by-component, following ToolRL's tool-name/parameter-name/parameter-value split).
  2. Generate a handful of real completions across deliberately varied prompt phrasings — some unambiguous, some vague on exactly one detail, some that could plausibly trigger a different but available tool.
  3. Score every completion both ways and look specifically for pairs where binary reward ties two completions that a human would clearly rank differently — that tie is the real, concrete cost of coarse reward, visible without training anything.
  4. If you have access to a small RL setup, try training on binary-only reward vs. the fine-grained version on the identical task, and watch specifically for RAGEN's named signature (reward variability collapse, entropy drop, gradient spikes) under whichever reward function is coarser — that's the real, connected risk this page's material predicts.

Scenario Check

Sources & further reading