Skip to content

Evaluating Agents

Hands-on

Full runnable recipe: evaluating-agents/ in the companion cookbook — outcome vs. trajectory grading, pass@k vs. pass^k on 5 real trials, and a capability-motivated prompt change that caught a real regression in its own grader.

TL;DR — quick revision
  • Outcome grading and trajectory grading answer different questions, and can disagree. Outcome asks "was the final answer right?" Trajectory asks "did it get there the intended way?" The Holistic Agent Leaderboard's own log audits caught agents "searching for the benchmark on HuggingFace instead of solving a task" — a case an outcome-only grader would have scored as a pass.
  • pass@k and pass^k measure opposite things. pass@k asks whether at least one of k tries succeeded (generous — useful for capability ceilings). pass^k asks whether every one of k tries succeeded (strict — the real reliability question, since a production agent gets one real try per request, not k with a human picking the best). τ-bench's own numbers show why the gap matters: GPT-4o's retail success rate falls from under 50% at pass@1 to under 25% at pass^8.
  • A real capability-motivated prompt change caught a genuine, clean regression — not in the model's reasoning, in the grader. Told to spell out numbers in words for a plausible accessibility reason, the model's arithmetic stayed completely correct ("one hundred divided by four equals twenty-five") — every task in a digit-matching regression suite still failed, because the grader, not the model, broke.
  • Infrastructure is a real, measurable confound in agent benchmarks. Running the identical model and benchmark across container configurations from strict to uncapped produced a 6-percentage-point swing (p < 0.01) on Terminal-Bench 2.0 — bigger than many reported leaderboard gaps — purely from resource allocation, with nothing about the model changing at all.
  • Capability evals and regression evals serve different jobs and shouldn't be conflated. A capability eval asks how good the agent could be — expensive, exploratory, run rarely. A regression eval asks whether a specific change broke something that used to work — cheap, narrow, run on every change. This page's own regression-suite experiment is exactly that second job, and it worked precisely because the suite was simple enough to run constantly.

Outcome vs. trajectory

Outcome grading checks the final result: did the agent produce the right answer, the right file diff, the right database state. It's cheap, unambiguous, and automatable — and it has a documented blind spot: an agent can reach the right outcome through a process nobody actually wanted. The Holistic Agent Leaderboard's own log-level audits found exactly this in the wild: agents "searching for the benchmark on HuggingFace instead of solving a task," or, in a different case, "misusing credit cards in flight booking tasks" — outcomes that a grader checking only the final state would have scored as passing, because nothing about how the result was reached was ever inspected. Trajectory grading closes that gap by checking the process itself — which tools got called, in what order, with what arguments — at the cost of being harder to define and more expensive to check than a single final-state comparison.

def _text(response, client, model):
    # client.messages.create(...) -- see agent_loop.py for the full call shape.
    ...


_REFERENCE_TOOL = {
    "name": "get_reference_fact",
    "description": "Look up a verified fact from the reference database, given a short topic string.",
    "input_schema": {"type": "object", "properties": {"topic": {"type": "string"}}, "required": ["topic"]},
}


def _run_with_optional_tool(question: str, fact_for_this_question: str, client, model) -> dict:
    """One turn: the model may call the reference tool or just answer.
    Returns the final answer and whether the tool was actually called --
    the trajectory, not just the outcome. The tool always returns the one
    fact relevant to this question, regardless of how the model phrases
    the lookup topic -- a fragile exact-string match on the model's own
    topic wording would test string-matching luck, not the mechanism."""
    response = client.messages.create(
        model=model, max_tokens=400, tools=[_REFERENCE_TOOL],
        messages=[{"role": "user", "content": f"Answer this question. You may call get_reference_fact to check the reference database if you need to.\n\n{question}"}],
    )
    tool_called = any(b.type == "tool_use" for b in response.content)
    if tool_called:
        tool_block = next(b for b in response.content if b.type == "tool_use")
        follow_up = client.messages.create(
            model=model, max_tokens=200, tools=[_REFERENCE_TOOL],
            messages=[
                {"role": "user", "content": f"Answer this question. You may call get_reference_fact to check the reference database if you need to.\n\n{question}"},
                {"role": "assistant", "content": response.content},
                {"role": "user", "content": [{"type": "tool_result", "tool_use_id": tool_block.id, "content": fact_for_this_question}]},
            ],
        )
        answer = _text(follow_up, client, model)
    else:
        answer = _text(response, client, model)
    return {"question": question, "answer": answer, "tool_called": tool_called}


def outcome_vs_trajectory_demo(client, model) -> dict:
    well_known = _run_with_optional_tool("What is the boiling point of water at sea level, in Celsius?", "100 degrees Celsius", client, model)
    well_known["outcome_correct"] = "100" in well_known["answer"]

    unguessable = _run_with_optional_tool("What year was the Halcyon Reach colony founded?", "2041", client, model)  # fictional, unguessable
    unguessable["outcome_correct"] = "2041" in unguessable["answer"]

    return {"well_known_fact": well_known, "unguessable_fictional_fact": unguessable}

A real run — both outcome and trajectory captured, for both a guessable and an unguessable fact

Input, well-known fact: "Answer this question. You may call get_reference_fact to check the reference database if you need to.\n\nWhat is the boiling point of water at sea level, in Celsius?" — a fact the model could plausibly answer correctly without ever calling the tool.

Output: the model called get_reference_fact anyway, then answered "The boiling point of water at sea level is 100 degrees Celsius." Outcome: correct. Trajectory: tool called, as intended.

Input, unguessable fictional fact: the identical instruction pattern, asking "What year was the Halcyon Reach colony founded?" — a fictional fact with no way to answer correctly except by calling the tool.

Output: the model called the tool, received the fact (2041), and answered "The Halcyon Reach colony was founded in 2041." Outcome: correct. Trajectory: tool called — the only way this outcome could have been correct at all.

No divergence in this real run — the model didn't take the shortcut of skipping verification even when it plausibly could have. That's a genuine, disclosed negative result, not a wasted test: this exact mechanism (checking whether the tool was actually called, not just whether the final answer was right) is precisely what would have caught the Holistic Agent Leaderboard's real documented cases if this model had behaved differently. Knowing the check doesn't fire on well-behaved runs is part of knowing it would fire on badly-behaved ones.

pass@k vs. pass^k

Both metrics run the same task k independent times, and they ask opposite questions of the same data. pass@k: did at least one of the k tries succeed? This is the metric behind "best-of-n" framings and capability ceilings — useful for asking "can this model do this at all," generous by construction. pass^k, introduced by τ-bench specifically to measure agent reliability, asks the stricter question: did every one of the k tries succeed? That's the metric that actually matches production reality, where an agent handling a real customer request doesn't get k attempts with a human picking the best one — it gets one, and pass^k is the probability that one attempt looks as good as all the others. τ-bench's own reported numbers make the gap concrete: GPT-4o's retail-domain success rate is under 50% at pass@1 (already not great), and — in Sierra's own description — falls to under 25% at pass^8, "a staggering 60% drop" from the single-shot number.

import re

_PROBABILITY_QUESTION = (
    "A committee of 3 people is chosen at random from a group of 5 men and 4 women. "
    "What is the probability that the committee has at least 2 women? Give your final answer "
    "as a simplified fraction, on its own line, in the form: Final answer: a/b"
)
_CORRECT_FRACTION = (17, 42)  # 34/84 simplified


def _check_probability_answer(text: str) -> bool:
    match = re.search(r"Final answer:\s*(\d+)\s*/\s*(\d+)", text)
    if not match:
        return False
    num, den = int(match.group(1)), int(match.group(2))
    return num * _CORRECT_FRACTION[1] == den * _CORRECT_FRACTION[0]


def pass_at_k_demo(client, model, k: int = 5) -> dict:
    """pass@k: did at least one of k independent trials succeed?
    pass^k: did EVERY one of k independent trials succeed? -- the real
    reliability question, since a production agent runs once per real
    request, not k times with a human picking the best try."""
    results = []
    for _ in range(k):
        response = client.messages.create(model=model, max_tokens=500, messages=[{"role": "user", "content": _PROBABILITY_QUESTION}])
        text = _text(response, client, model)
        results.append({"answer": text, "correct": _check_probability_answer(text)})

    successes = sum(1 for r in results if r["correct"])
    pass_at_k = successes >= 1
    pass_hat_k = successes == k
    return {"k": k, "successes": successes, "pass_at_k": pass_at_k, "pass_hat_k": pass_hat_k}

A real run — 5 independent trials, the same question, every answer shown

Input, identical across all 5 trials: "A committee of 3 people is chosen at random from a group of 5 men and 4 women. What is the probability that the committee has at least 2 women? Give your final answer as a simplified fraction, on its own line, in the form: Final answer: a/b" Ground truth (computed independently): 17/42.

Output — all 5 trials, real final lines: trial 1 through 5 each worked through the same two cases (exactly 2 women, exactly 3 women) via combinations — C(4,2)×C(5,1) = 30 and C(4,3)×C(5,0) = 4, giving 34/84, simplified to 17/42 — and every trial ended with the identical line: Final answer: 17/42.

5/5 correct. pass@5 = true. pass^5 = true. No gap observed on this specific task — an honest result, not a disappointing one: this combinatorics problem, at this difficulty, showed perfect reliability for Claude Haiku 4.5 across 5 real trials. That doesn't contradict τ-bench's reported 50%-to-25% drop; it shows that the gap between pass@k and pass^k is a property of how much genuine variance a specific task has for a specific model, not something inherent to the metrics — τ-bench's tasks (multi-turn, tool-using, policy-constrained customer service) have far more room for a single trial to go sideways than one well-structured probability question does.

Capability vs. regression suites

Anthropic's own framing draws a clean, practical line: a capability eval asks how good the agent could be — often exploratory, expensive to run, and used to guide real development decisions. A regression eval asks a narrower, cheaper question: did a specific change break something that used to work? It has to be fast and stable enough to run on every change, the same job a unit-test suite does in ordinary software. Conflating the two is a real, common mistake: a capability eval's occasional, expensive, deep-dive numbers aren't a substitute for a small, boring, constantly-run regression suite — and a regression suite that never catches anything isn't necessarily a badly-designed one, since "still passing" is the correct default outcome most of the time it runs.

_REGRESSION_SUITE = [
    ("What is 2 + 2?", "4"),
    ("What is 10 * 5?", "50"),
    ("What is 100 / 4?", "25"),
    ("What is 6 squared?", "36"),
]

_BASELINE_SYSTEM = "Answer the arithmetic question directly."
# Motivated by a real, common driver of silent regressions: a formatting
# change made for a genuinely good reason (here, an accessibility-style
# guideline) that a downstream automated grader -- built when answers were
# plain digits -- was never updated to handle.
_CAPABILITY_MOTIVATED_SYSTEM = (
    "Express all numeric answers in words, not digits (e.g. 'forty-two', not '42'), as required "
    "by the new accessibility guideline. This applies to every number in your response, no exceptions."
)


def _run_regression_suite(system: str, client, model) -> list:
    results = []
    for question, expected in _REGRESSION_SUITE:
        response = client.messages.create(model=model, max_tokens=500, system=system, messages=[{"role": "user", "content": question}])
        text = _text(response, client, model)
        passed = expected in text
        results.append({"question": question, "expected": expected, "answer": text, "passed": passed})
    return results


def capability_vs_regression_demo(client, model) -> dict:
    baseline_results = _run_regression_suite(_BASELINE_SYSTEM, client, model)
    new_results = _run_regression_suite(_CAPABILITY_MOTIVATED_SYSTEM, client, model)
    return {
        "baseline": {"results": baseline_results, "all_passed": all(r["passed"] for r in baseline_results)},
        "capability_motivated_change": {"results": new_results, "all_passed": all(r["passed"] for r in new_results)},
    }

A real run — a plausible capability-motivated change, and a real, clean regression

Input, baseline condition: 4 trivial arithmetic tasks ("What is 2 + 2?", "What is 10 * 5?", "What is 100 / 4?", "What is 6 squared?"), plain system prompt: "Answer the arithmetic question directly." Output: all 4 answers contained the correct literal digits (e.g. "100 / 4 = 25") — 4/4 passed an automated grader checking for the expected digit string in the reply.

Input, capability-motivated condition: the identical 4 tasks, system prompt changed to: "Express all numeric answers in words, not digits (e.g. 'forty-two', not '42'), as required by the new accessibility guideline. This applies to every number in your response, no exceptions." — a plausible, real reason to change a system prompt.

Output: "Two plus two equals four." … "Ten multiplied by five equals fifty." … "One hundred divided by four equals twenty-five." … "Six squared is thirty-six." Every answer is completely correct arithmetic — and all 4 tasks failed the same automated grader, because none of "four," "fifty," "twenty-five," "thirty-six" contains the literal digit string the grader was checking for.

This is a real, clean regression, and it's worth being precise about what actually regressed: not the model's reasoning — the grader. The accessibility-motivated change was a reasonable thing to want, and it silently broke an automated check that assumed a format nobody thought to re-verify. This is exactly the failure mode a regression suite exists to catch cheaply, before a much more expensive capability eval — or a real user — catches it instead.

Infrastructure noise

One more source of variance in agent evals doesn't come from the model or the task at all: the hardware the eval runs on. Anthropic ran the identical model, harness, and task set (Terminal-Bench 2.0) across six container resource configurations, from strict 1x enforcement to fully uncapped — nothing about the agent changed between runs, only how much CPU and memory it had available. The result: a 6-percentage-point gap (p < 0.01) between the most- and least-resourced configurations — larger than many reported gaps between competing systems on public leaderboards. The mechanism is largely infrastructure-level failures, not reasoning failures: the infra error rate itself fell from 5.8% at 1x headroom to 2.1% at 3x (p < 0.001), down to 0.5% uncapped. A cross-check on SWE-bench, varying RAM up to 5x baseline across 227 problems, found a smaller but still real effect (+1.54 percentage points at 5x versus 1x) — smaller because SWE-bench's tasks are less resource-intensive than Terminal-Bench 2.0's, which is itself a useful finding: the size of the infrastructure confound depends on the task's own resource profile, not just on infrastructure existing. Anthropic's own recommendation follows directly: benchmark maintainers should publish recommended resource specs and enforcement methodology, and treat resource configuration "as a first-class experimental variable, documented and controlled" — the same way a rigorous experiment controls for any other variable that could explain a result. This page didn't reproduce the experiment itself (it needs real container orchestration across multiple resource tiers, not a lightweight script), but the numbers above are real, cited, and worth internalizing before trusting a close leaderboard gap: a 2-point lead could be silicon, not intelligence.

Interview angle

Weak answer to "how would you evaluate a new agent before shipping it": "Run it against a benchmark and check the pass rate." This treats evaluation as one number from one run, and it can't explain any of this page's own real findings — not the outcome/trajectory gap (a single pass rate can't tell you whether the agent got there honestly), not pass@k vs. pass^k (a single run can't distinguish "usually works" from "always works"), and not the regression-suite finding (a one-off capability number wouldn't have caught a grader silently breaking on a routine prompt change).

Strong answer: evaluation is at least three separate questions, each needing a different setup. What's the agent capable of, at its ceiling — a capability eval, run occasionally, expensive, exploratory. Did a specific change break something that used to work — a regression suite, run constantly, cheap, narrow, and this page's own real trace shows it catching a failure a capability eval wouldn't even have been looking for. And is the agent's success outcome-real or trajectory-real — checked by actually inspecting the process, not just the final state, because the literature has real, documented cases (searching for a benchmark instead of solving the task; misusing a payment tool) where those two disagree. Layered under all three: how many trials is the reported number even based on, and was the hardware held constant — a single pass@1 number on unspecified infrastructure is not evidence, it's a claim.

Follow-up to expect: "if your own pass@k and pass^k came out identical, doesn't that mean the distinction doesn't matter in practice?" No — it means this specific task, at this difficulty, for this model, had low variance in this run. τ-bench's own published numbers (a 60% relative drop from pass@1 to pass^8 in a genuinely harder, multi-turn, tool-using domain) are the evidence the distinction matters in general; this page's clean result is evidence that the size of the gap is task-dependent, which is itself the useful, precise version of the claim — not "pass@k and pass^k always diverge," but "check both, because whether they diverge tells you something real about the task."

Build it yourself — 30 minutes

  1. Pick a task where the right way to get there is checkable — a tool that must be called, a specific sequence of steps. Run it once, then grade it two ways: outcome only (is the final answer right?) and trajectory (did it take the intended path?). Deliberately try a version of the prompt that makes skipping the intended path plausible, and see if the two graders ever disagree.
  2. Run one task 5 times with real API calls (not a mock). Compute pass@5 and pass^5 from the real results, not an assumption. If they're identical, that's information about the task's variance, not a failed experiment.
  3. Build a tiny regression suite — 3 or 4 trivial tasks with an exact-match or digit-match grader. Make one plausible, unrelated system-prompt change (a tone requirement, a formatting rule) and re-run the suite. Check whether it still passes — and if it doesn't, check whether the model regressed or the grader did.
  4. If you have access to adjustable compute (a container resource limit, a local model's thread count), try running the identical eval at two different resource levels and see whether the score moves at all before trusting a close comparison between two systems.

Scenario Check

Sources & further reading