Skip to content

Cost and Latency

Hands-on

Full runnable recipe: cost-latency/ in the companion cookbook — a real, measured transcript-growth repro plus a real confidence-routing test.

TL;DR — quick revision
  • Naive agent loops don't cost N times as much as a single call — they cost roughly N² times as much. Every turn resends the full accumulating transcript, so a real, documented formula applies: Total = N×S + u×N(N+1)/2 + r×N(N-1)/2 — the N(N+1)/2 triangular-number term is the trap. A real, worked example: a 20-step loop generating 1,000 tokens/step produces 210,000 cumulative input tokens, not the 20,000 a naive per-step estimate would suggest.
  • A real repro confirmed the shape directly, and measured what windowing buys: a naive 6-file tool loop's per-turn input tokens grew by a near-constant ~631 tokens each turn (665→1290→1921→2552→3183→3814→4445) — that constant per-turn increase is exactly what makes the cumulative total quadratic. A windowed version (only the last 2 tool results sent in full) flattened that increase to ~121 tokens/turn by the end, for a real 28.3% cumulative reduction — a gap that only widens with more turns.
  • Model routing has a real, large lever: current published pricing puts small and large tiers roughly 5-10x apart per token (Haiku 4.5 at $1/$5 per million vs. Sonnet 5 at $2/$10 per million, input/output), and a real repro confirmed the upside directly — Haiku matched Sonnet's accuracy on a 10-question batch at ~44% of the real dollar cost.
  • But the routing mechanism itself has a real, sharp limitation: asking a model to self-report its own confidence is not a reliable way to catch its own mistakes. A real repro found the smaller model reported high confidence on every question, including the one it got wrong — confidently, not hesitantly. This is a concrete, measured reason production routing methods lean on a different signal (like model-internal token-probability confidence) instead of self-report alone.

The real shape of the cost trap: quadratic, not linear

The intuitive mental model for agent cost is "N turns, N times the cost of one turn." That's wrong for the common case, and a real, documented formula shows exactly why: Total = N×S + u×N(N+1)/2 + r×N(N-1)/2, where S is the fixed system-prompt size, u is new input added per turn (a user message or tool result), r is output tokens per turn, and N is the total number of turns. The N(N+1)/2 term is a triangular number — it grows quadratically in N — because a naive loop resends the entire accumulating transcript on every single call, meaning turn 1's content gets rebilled on turns 2 through N, turn 2's content gets rebilled on turns 3 through N, and so on.

The real, worked scale of this: a 20-step loop where each step generates 1,000 tokens produces 210,000 cumulative input tokens — not the 20,000 a flat per-step estimate would suggest. A real, separate worked example for a 10-step file-reading agent (Sonnet-tier pricing, 8,000 input tokens/step): a naive loop costs 43.3x a single-pass baseline; a constrained, windowed version cuts that to roughly 29x — real, substantial savings, but still nowhere near 1x, because some rebilling is unavoidable once a loop spans multiple turns at all.

Repro: measuring the quadratic shape directly

The block below defines the fixed 6-file task and a naive tool-calling loop that resends the entire growing transcript every turn — the setup that produces the per-turn token counts reported below.

_PADDING = "# padding line to simulate a real config file with surrounding noise\n" * 40
_VALUES = [17, 42, 8, 23, 91, 5]
_FILES = {
    f"config_{i}.txt": f"# Service configuration shard {i} of 6\n{_PADDING}CONFIG_VALUE={value}\n"
    for i, value in enumerate(_VALUES, start=1)
}
_TRUE_SUM = sum(_VALUES)

_TRANSCRIPT_TASK = (
    "You have access to 6 configuration files: config_1.txt through config_6.txt. "
    "Read all of them using the read_file tool, then report the SUM of the CONFIG_VALUE "
    "found across all 6 files as a single final number, on its own line, prefixed with 'SUM='. "
    "Important: call read_file for exactly ONE file per response, and wait for that file's "
    "result before requesting the next one -- do not request multiple files in a single response."
)

The block below adds a windowed variant of that same loop, keeping only the last 2 tool results sent in full, to measure how much of that per-turn growth a simple context constraint actually removes.

def _build_sent_messages(full_messages: list, window: int | None) -> list:
    """The windowing logic under test -- pure, no API calls. If window is None, sends the
    full transcript unchanged (the naive condition). Otherwise, any tool_result block more
    than `window` tool-results back gets its content replaced with a short placeholder,
    while the message structure (roles, tool_use ids) stays intact so the API still accepts
    the conversation."""
    if window is None:
        return full_messages

    tool_result_positions = [i for i, m in enumerate(full_messages) if m["role"] == "user" and isinstance(m["content"], list)
                              and any(b.get("type") == "tool_result" for b in m["content"])]
    keep_from = set(tool_result_positions[-window:]) if tool_result_positions else set()

    sent = []
    for i, m in enumerate(full_messages):
        if i in tool_result_positions and i not in keep_from:
            sent.append({
                "role": "user",
                "content": [
                    {"type": "tool_result", "tool_use_id": b["tool_use_id"], "content": "[older tool result omitted to save context]"}
                    if b.get("type") == "tool_result" else b
                    for b in m["content"]
                ],
            })
        else:
            sent.append(m)
    return sent

The windowing logic is deliberately simple: keep the message structure intact (every tool_use still has a matching tool_result, so the API still accepts the conversation) but replace old tool results' content with a short placeholder once they fall outside the window.

A real run — naive vs. windowed, identical 6-file task, real per-turn token counts

naive (full transcript resent every turn), real per-turn input_tokens: 665, 1290, 1921, 2552, 3183, 3814, 4445. Each turn costs about 631 tokens more than the one before it — a constant per-turn increase, which is exactly what a quadratic cumulative total looks like turn by turn. Real cumulative total: 17,870 input tokens across 7 turns.

windowed_last_2 (only the 2 most recent tool results sent in full), real per-turn input_tokens: 665, 1300, 1934, 2051, 2168, 2285, 2406. Nearly identical to naive for the first 3 turns — the window hasn't started dropping anything yet, since fewer than 3 tool results exist. From turn 4 on, the per-turn increase flattens to about 121 tokens/turn. Real cumulative total: 12,809 input tokens — a real 28.3% reduction, and the gap widens every additional turn past this one.

Both conditions reached the correct final answer (SUM=186) — windowing here cost nothing in correctness, because the task's final step only needed the sum, not the individual file contents still visible earlier in the untouched transcript.

Routing: a real, positive result and a real, sharp limitation, in the same run

The block below defines the routing tool itself — a Haiku-first call that must also report its own confidence — and the ten real questions it gets tested against.

_ANSWER_TOOL = [{
    "name": "submit_answer",
    "description": (
        "Submit your final answer and your genuine confidence in it. Mark confidence as 'low' "
        "whenever the question involves multi-step arithmetic, a classic-seeming riddle that might "
        "have a non-obvious twist, or any reasoning chain where a careless mistake is plausible -- "
        "not just when you're unsure of a fact. Reserve 'high' for cases you'd bet money on."
    ),
    "input_schema": {
        "type": "object",
        "properties": {
            "answer": {"type": "string", "description": "The final answer, as short as possible."},
            "confidence": {"type": "string", "enum": ["high", "low"], "description": "Your genuine, calibrated confidence -- not a default."},
        },
        "required": ["answer", "confidence"],
    },
}]

The block below is the loop that actually runs those ten questions two ways — always-Sonnet versus Haiku-with-escalation — and tallies the real cost and accuracy numbers reported next.

def _ask(client, model: str, question: str) -> dict:
    response = client.messages.create(
        model=model, max_tokens=1024, tools=_ANSWER_TOOL, tool_choice={"type": "tool", "name": "submit_answer"},
        messages=[{"role": "user", "content": question}],
    )
    block = next(b for b in response.content if b.type == "tool_use")
    return {"answer": block.input["answer"], "confidence": block.input.get("confidence", "high")}


def _is_correct(item: dict, answer: str) -> bool:
    return item["answer"].lower() in answer.lower()


def routing_demo(client, model_small: str, model_large: str, questions: list) -> dict:
    always_large = []
    routed = []

    for item in questions:
        large_result = _ask(client, model_large, item["q"])
        always_large.append({**large_result, "correct": _is_correct(item, large_result["answer"])})

        small_result = _ask(client, model_small, item["q"])
        if small_result["confidence"] == "low":
            escalated = _ask(client, model_large, item["q"])
            routed.append({**escalated, "escalated": True})
        else:
            routed.append({**small_result, "escalated": False})

    return {"always_large": always_large, "routed": routed}

Ten real questions — some trivial, some genuinely tricky (a bat-and-ball problem, a box-relabeling logic puzzle, a compound-discount trap, and a "strawberry" spelling trap) — answered two ways: every question straight to Sonnet 5, versus Haiku 4.5 first with an explicitly-calibrated confidence field, escalating to Sonnet 5 only on a low reading.

A real run, unforced — real dollar cost using current published pricing

Cost and accuracy: always_sonnet spent $0.026866 for 9/10 correct. routed spent $0.011843 for the identical 9/10 correct — Haiku 4.5 alone matched Sonnet 5's real accuracy on this batch at ~44% of the real dollar cost. A genuine, positive result: the cheap model was fully adequate here, not a compromise.

But escalated_count: 0. Haiku reported confidence: "high" on all 10 questions — including the one it got wrong. The miss was the "strawberry" spelling trap; Haiku confidently answered "Yes" (there are two consecutive 'r's), matching Sonnet's own real miss on the identical question, with identical confidence. This held even after one deliberate adjustment made specifically to give the routing mechanism a fairer shot: two additional hard, multi-step questions were added, and the tool's own description was rewritten to explicitly instruct calibrated self-assessment — "mark confidence as low whenever the question involves multi-step arithmetic... not just when you're unsure of a fact." Confidence stayed high across the board regardless.

Why self-reported confidence isn't the mechanism production routers actually rely on

The real, sharper finding is in that null result, not despite it: a model that's about to be wrong isn't reliably aware that it's about to be wrong, so asking it to say so in words is not a dependable safety net. This is a concrete, measured version of a real, documented design choice: confidence-guided routing methods like STEER (arXiv 2511.06190) use model-internal confidence — derived from the smaller model's own output-token probabilities — rather than a self-reported natural-language field, specifically because that internal signal doesn't depend on the model correctly narrating its own uncertainty. This recipe's self-report mechanism is the practical fallback available through a hosted API that doesn't expose token-level logprobs the way some routing methods assume — Anthropic's Messages API is one such case — and the real result here is exactly why that gap matters: the cost benefit of routing was real and measured, but the safety benefit of routing (catching the small model's mistakes before they ship) did not materialize from self-report alone.

What this means in practice

Two separate, real levers, not one: windowing/context-constraining attacks the quadratic term directly — it doesn't change which model answers, it changes how much of the transcript gets rebilled each turn, and the real measured gap (28.3% here, growing with more turns) compounds the longer a loop runs. Routing attacks the per-token price — a real, large lever (5-10x per token on current pricing) that this run's cost numbers confirm directly. But routing's safety case — using it to keep the expensive model's judgment on anything genuinely risky — needs a routing signal that doesn't rely on the small model accurately reporting its own uncertainty, because this run's own real result shows that signal failing exactly when it mattered most: on the one question the small model actually got wrong.

Interview angle

Weak answer to "how would you cut this agent's API cost?": "Switch to a cheaper model." This page's own repro shows why that's incomplete on its own — the quadratic transcript-growth term applies regardless of which model answers, and a cheap model rebilling an ever-growing transcript still hits the same N(N+1)/2 wall, just at a lower per-token price.

Strong answer: separate the two real levers explicitly. Attack the quadratic term with context management (windowing, summarization, or a fresh sub-agent per bounded subtask) — this page's own real numbers show a measured, compounding reduction. Attack the per-token price with routing — but be precise about what routing actually buys: this page's own real result got the cost benefit (44% of cost, identical accuracy) and a clean demonstration that the naive escalation trigger (self-reported confidence) never fired, even on the one question that needed it. A candidate who names that distinction — cost savings from routing is not the same guarantee as safety from routing, unless the escalation signal is actually reliable — is demonstrating real, measured understanding, not a generic cost-optimization checklist.

Follow-up to expect: "if self-reported confidence doesn't work, what would?" A real, honest answer from this page's own material: either a deterministic, non-LLM check on the output (when the task allows one, the same structural-versus-behavioral distinction Guardrails and Human-in-the-Loop makes about approval gates), or a model-internal confidence signal like STEER's token-probability approach — which this page's own repro couldn't test directly, since it's not exposed through the Messages API used throughout this project, a real, current constraint worth naming rather than glossing over.

Build it yourself — 30 minutes

  1. Pick a real multi-turn tool loop you already have (or the file-reading task above). Instrument it to print the real usage.input_tokens from every API response, and sum them across the full run.
  2. Run it naive (full transcript every turn) and note the per-turn deltas — confirm they're roughly constant, which is what produces the quadratic cumulative total.
  3. Add a real context constraint (a window, a summarizer, or dropping tool results once superseded) and re-run the identical task. Compare real cumulative totals, not just per-call estimates.
  4. Separately, take a batch of real task instances at varying difficulty and try confidence-based routing with your cheapest adequate model. Check specifically whether the model's self-reported confidence was ever low on the instances it actually got wrong — that's the real test of whether the mechanism is doing its stated job, not just whether it saved money on average.

Scenario Check

Sources & further reading