Skip to content

Tool Design

Hands-on

Full runnable recipe: tool-design/ in the companion cookbook — the same fictional task-tracker data exposed through three tool surfaces, measured on real call count and token count, plus terse vs. actionable tool-error messages compared on real recovery quality.

TL;DR — quick revision
  • A tool description is a prompt, not documentation — it steers behavior the same way a system prompt does. Anthropic reports that precise tool-description refinements (nothing else) took Claude Sonnet 3.5 to state-of-the-art on SWE-bench Verified.
  • Consolidated, purpose-built tools beat many granular endpoint wrappers on both calls and tokens, measured: on the identical question, a single get_blocked_tasks_with_reasons tool took 2 calls and 1,563 tokens; exposing list_tasks + list_comments separately and letting the model compose them took 3 calls and 3,079 tokens.
  • Code execution's real advantage didn't show up as a clean win on one fixed question — it tied endpoint wrappers on call count (3 each) here, though it used fewer tokens (2,951 vs. 3,079). Its actual case is avoiding a combinatorial explosion of purpose-built tools across many different possible query shapes, not raw efficiency on one you already anticipated.
  • A real error-message comparison's finding wasn't about retry count — both a terse and an actionable validation error took the model exactly 2 calls to recover. What differed was the value it recovered to: the terse error (no information) got a generic, disconnected guess; the actionable error (stating the valid options) got a genuinely closer match to what the user actually meant.
  • Interface design measurably matters independent of the underlying model — SWE-agent's own contribution wasn't a better model, it was a custom Agent-Computer Interface, and the paper reports this got a non-interactive baseline's pass@1 up to 12.5% on SWE-bench.

Schema as prompt

The tool schema — its name, its description, its parameter names and descriptions — is not metadata the model consults once and moves past. It's read on every turn a tool might be relevant, the same way a system prompt is, and it steers behavior with the same weight. Anthropic's own guidance states the practical test plainly: describe a tool the way you'd describe it to a new hire, and make context that's obvious to you but invisible to the model explicit in the text. The reported result of doing this rigorously: refining tool descriptions alone — no model change, no architecture change — took Claude Sonnet 3.5 to state-of-the-art performance on SWE-bench Verified.

Ambiguous parameter names (user instead of user_id) and thin descriptions don't just risk a wrong call — they cost tokens and turns recovering from one, which the rest of this page measures directly rather than asserting.

_TASKS = {
    "T-101": {"title": "Fix login redirect bug", "status": "blocked",
              "comments": ["Reproduced on staging.", "Root cause looks like a stale session cookie.",
                            "Blocked: waiting on infra team to rotate the session-signing key before we can test the fix."]},
    "T-102": {"title": "Add dark mode toggle", "status": "in_progress",
              "comments": ["Design mockups approved.", "Implementing CSS variables now."]},
    "T-103": {"title": "Migrate billing service to new queue", "status": "blocked",
              "comments": ["Queue capacity looks fine.", "Blocked: legal needs to sign off on the new data-retention policy before migration can start."]},
    "T-104": {"title": "Write onboarding docs", "status": "done",
              "comments": ["Draft complete.", "Reviewed and merged."]},
    "T-105": {"title": "Optimize search index rebuild", "status": "in_progress",
              "comments": ["Benchmarked current rebuild time: 40 minutes."]},
    "T-106": {"title": "Deprecate legacy export endpoint", "status": "blocked",
              "comments": ["Usage dropped to near zero.", "Blocked: one internal dashboard still depends on it, waiting for that team to migrate."]},
}

TOOL_QUESTION = "Which tasks are currently blocked, and what's the most recent blocking reason mentioned in their comments? List each task's title and that reason."


def list_tasks() -> list:
    return [{"id": tid, "title": t["title"], "status": t["status"]} for tid, t in _TASKS.items()]


def list_comments(task_id: str) -> list:
    if task_id not in _TASKS:
        return {"error": f"No task '{task_id}'"}
    return _TASKS[task_id]["comments"]

The agent-computer interface: SWE-agent's finding

SWE-agent's paper frames a claim worth stating precisely, because it's easy to round off into "better prompting": language model agents are "a new category of end users with their own needs and abilities," and would benefit from interfaces built specifically for them — not the same interfaces built for humans, repurposed. The paper's own custom Agent-Computer Interface (ACI) — file navigation and editing commands, test execution, purpose-built for an LM to use reliably — reports a 12.5% pass@1 on SWE-bench, "far exceeding" the previous state of the art with non-interactive language models on the same benchmark family (Yang et al., 2024). The interface, not a stronger model, is the paper's contribution. That's the same idea underneath every comparison on this page: the surface an agent gets isn't a thin wrapper around "the real API" — it's a genuine design decision with a measurable effect on the outcome.

Endpoint wrappers vs. consolidated vs. code execution, measured

Three ways to expose the same fictional task data to the same model, for the same question: "Which tasks are currently blocked, and what's the most recent blocking reason mentioned in their comments?"

Endpoint wrappers — one tool per low-level operation, composition left entirely to the model:

ENDPOINT_TOOLS = [
    {"name": "list_tasks", "description": "List all tasks with id, title, and status.", "input_schema": {"type": "object", "properties": {}}},
    {"name": "list_comments", "description": "List all comments for a task, oldest first. Example: list_comments(task_id='T-101').",
     "input_schema": {"type": "object", "properties": {"task_id": {"type": "string"}}, "required": ["task_id"]}},
]
# Model must: list all tasks, notice which are blocked, then call
# list_comments separately for each blocked one -- every intermediate
# result (including the 3 non-blocked tasks' data) passes through context.

Consolidated — one purpose-built tool that does the filtering server-side:

def get_blocked_tasks_with_reasons() -> list:
    return [{"id": tid, "title": t["title"], "latest_comment": t["comments"][-1]} for tid, t in _TASKS.items() if t["status"] == "blocked"]


CONSOLIDATED_TOOLS = [
    {"name": "get_blocked_tasks_with_reasons",
     "description": "Return every currently blocked task, each with its title and its most recent comment (the blocking reason).",
     "input_schema": {"type": "object", "properties": {}}},
]
# One purpose-built tool does the filtering server-side -- the model never
# sees the non-blocked tasks or any comment except the one it needs.

Code execution — the model writes code against a small library instead of calling named tools one at a time:

import contextlib
import io


def execute_python(code: str) -> str:
    """A minimal, non-production sandbox: restricted builtins, only
    list_tasks/list_comments exposed, stdout captured as the tool result.
    Real production code-execution needs process isolation and timeouts --
    this is illustrative of the mechanism, not a deployable sandbox."""
    safe_globals = {"__builtins__": {"len": len, "range": range, "print": print, "sorted": sorted, "enumerate": enumerate}}
    safe_locals = {"list_tasks": list_tasks, "list_comments": list_comments}
    buffer = io.StringIO()
    try:
        with contextlib.redirect_stdout(buffer):
            exec(code, safe_globals, safe_locals)
    except Exception as e:
        return f"Error executing code: {e}"
    return buffer.getvalue() or "(no output -- did you print the result?)"


CODE_EXEC_TOOLS = [
    {"name": "execute_python",
     "description": (
         "Run Python code against a small library: list_tasks() -> [{id, title, status}], "
         "list_comments(task_id) -> [str, ...] (oldest first). Print whatever you need as the "
         "final result -- only what you print is returned to you, so filter and summarize in code "
         "rather than printing raw data."
     ),
     "input_schema": {"type": "object", "properties": {"code": {"type": "string"}}, "required": ["code"]}},
]
# The model writes the filtering logic itself; raw comment text for
# non-blocked tasks never has to become tokens in its context at all.

A real run — the exact question, three surfaces, real call and token counts

Input, identical for all three surfaces: "Which tasks are currently blocked, and what's the most recent blocking reason mentioned in their comments? List each task's title and that reason." Ground truth (computed independently, not by the model): 3 blocked tasks — Fix login redirect bug (waiting on infra to rotate the session-signing key), Migrate billing service to new queue (waiting on legal sign-off on data-retention policy), Deprecate legacy export endpoint (an internal dashboard still depends on it).

Surface Calls Total tokens
Endpoint wrappers 3 3,079
Consolidated 2 1,563
Code execution 3 2,951

All three surfaces got the correct answer, Claude Sonnet 5 throughout. Consolidated won outright — half the tokens of endpoint wrappers, the fewest calls. Code execution's result is the honest, unglamorous one: it tied endpoint wrappers on calls (3 each) and only modestly beat it on tokens (2,951 vs. 3,079) — nowhere near a clean win on this specific question. That's not a failure to report quietly; it's the actual finding. Code execution's real case isn't "cheaper on any one query" — it's that a consolidated surface only stays cheap if you correctly anticipated the query shape in advance and built a bespoke tool for it. Ask a differently-shaped question tomorrow and a fixed set of consolidated tools may not cover it, while code execution's model-written filtering logic generalizes to a new shape for free. One fixed question can't demonstrate that scaling argument — it would take many different query shapes, run against the same three surfaces, to show it honestly, which is future work, not something to claim from this run.

Errors as observations

A tool's error message is also part of its interface, read by the model as an observation it has to act on — not a log line for a human. Anthropic's own guidance is specific: errors should give "specific and actionable improvements, rather than opaque error codes or tracebacks."

import re

_VALID_PRIORITIES = {"low", "medium", "high", "urgent"}
_DATE_RE = re.compile(r"^\d{4}-\d{2}-\d{2}$")


def _validate_create_task(title: str, priority: str, due_date: str, actionable: bool) -> dict:
    if priority not in _VALID_PRIORITIES:
        if actionable:
            return {"error": f"Invalid priority '{priority}'. Must be exactly one of: low, medium, high, urgent."}
        return {"error": "ValidationError: priority"}
    if not _DATE_RE.match(due_date):
        if actionable:
            return {"error": f"Invalid due_date '{due_date}'. Must be in YYYY-MM-DD format, e.g. 2026-10-02."}
        return {"error": "ValidationError: due_date"}
    return {"result": f"Created task '{title}' (priority={priority}, due={due_date})"}


# Two conditions, same tool, same request -- only the validation error's
# wording changes. The terse version names what field is wrong; the
# actionable version also states the valid values or the expected format.

A real run — same request, two error conditions, both shown in full

Input, identical for both conditions: "Create a task to fix the login bug, due next Friday, pretty urgent. You can't ask me follow-up questions right now — call create_task directly. For the priority field, pass the user's own words exactly as given ('pretty urgent'); make your best guess for the date." (Deliberately forcing a first attempt with an invalid priority value, so there's a real error to observe and recover from.)

Terse condition — first call: {"title": "fix the login bug", "priority": "pretty urgent", "due_date": "2025-01-10"} → {"error": "ValidationError: priority"}. Second call, recovering: {"title": "fix the login bug", "priority": "high", "due_date": "2025-01-10"} → succeeded. 2 calls.

Actionable condition — first call: {"title": "fix the login bug", "priority": "pretty urgent", "due_date": "2025-01-10"} → {"error": "Invalid priority 'pretty urgent'. Must be exactly one of: low, medium, high, urgent."}. Second call, recovering: {"title": "fix the login bug", "priority": "urgent", "due_date": "2025-01-10"} → succeeded. 2 calls.

Both conditions converged in exactly the same number of calls — the actionable error didn't make recovery faster. What it changed is which value the model recovered to: given no information about what's valid, the model played it safe and picked "high", a generic middle value with no real connection to "pretty urgent." Given the actual list of valid options, it picked "urgent" — the option that genuinely matches what the user said. An honest, more precise version of "better errors help": not always fewer turns, but a better answer at the end of the same number of turns.

Tool evals: measuring what actually matters

Everything above is a small version of a practice Anthropic reports running at production scale: building held-out evaluation sets for real tool integrations (Slack, Asana) and iterating tool descriptions and shapes against measured accuracy, the same way one would iterate a prompt. Separately, Anthropic's advanced tool-use work reports concrete numbers for related techniques: a Tool Search mechanism (only loading tool definitions relevant to a query instead of the full library) cutting token usage by 85% and moving one internal MCP evaluation's accuracy from 49% to 74% on Opus 4 (79.5% to 88.1% on Opus 4.5); Programmatic Tool Calling cutting tokens by 37% on a complex research task; and worked examples in a tool's description (not just its schema) moving accuracy on tricky parameter handling from 72% to 90%. The pattern across all of it, including this page's own three small real numbers above: tool design is something you measure and iterate, the same way you would a prompt or a retrieval pipeline — not something you get right once by intuition and leave alone.

Interview angle

Weak answer to "how would you decide how many tools to expose to an agent": "Fewer tools is always better, since it's less for the model to get confused by." This states a real, common pressure but not the actual trade-off, and it can't explain this page's own real result — consolidated (1 tool) beat endpoint wrappers (2 tools) on this run, but code execution (also 1 tool) didn't beat endpoint wrappers at all on calls.

Strong answer: the real axis isn't tool count, it's whether the tool surface matches the shape of the questions actually being asked. A single consolidated tool is cheap exactly when you can predict the query shape in advance and build the tool to answer it directly — which is precisely what this page's real run showed for one fixed question. The moment the range of possible queries gets wide or unpredictable, a fixed set of purpose-built tools either grows without bound (defeating the "fewer tools" instinct) or fails to cover a shape nobody anticipated — which is the actual argument for code execution and endpoint-style primitives: not that they're cheaper per call, but that they compose to answer questions nobody pre-built a tool for.

Follow-up to expect: "so should every agent just use code execution instead of purpose-built tools?" No — this page's own numbers are the counter-evidence: code execution didn't win on this specific, single, well-anticipated query. The real decision criterion is the actual diversity of queries the agent will face in production — narrow and predictable favors consolidated tools measured against real evals; broad and unpredictable favors giving the model primitives and letting it compose them, accepting a per-query cost that's sometimes higher than a tool you could have pre-built, in exchange for coverage you couldn't have pre-built for everything.

Build it yourself — 30 minutes

  1. Pick a small, fictional dataset and a fixed question against it — something with an answer you can compute independently, so you can check correctness, not just "did it respond."
  2. Build the endpoint-wrapper surface first: one tool per basic operation, no composition helpers. Run the real question, record the real call count and token usage from your API response's usage field.
  3. Build one consolidated tool purpose-built for that exact question. Run the same question again, compare the real numbers against step 2 — don't estimate them.
  4. Pick one tool in your surface and deliberately give it a validation error with no information (just an error type). Trigger it, then rewrite the error to name the actual valid values or expected format, and trigger it again with the identical ambiguous request. Compare not just how many turns each takes, but what value the model actually recovers to.

Scenario Check

Sources & further reading