Skip to content

Observability and Debugging

Hands-on

Full runnable recipe: observability-debugging/ in the companion cookbook — a real, spec-shaped OpenTelemetry GenAI trace that makes a hidden tool bug diagnosable without re-running anything.

TL;DR — quick revision
  • Agent failures don't look like failures from the outside — that's the actual problem this whole track exists to solve. The real, current framing: "When your AI agent returns a confidently wrong answer, your monitoring sees a successful 200 response." (Jamie Mallers, OneUptime, 2026-03-28) — traditional health checks (did the call error, did it time out) are structurally blind to an agent that completes cleanly and is simply wrong.
  • OpenTelemetry's real GenAI semantic conventions give this a standard shape, not a bespoke one: span names follow {gen_ai.operation.name} {gen_ai.tool.name} (e.g. execute_tool get_daily_active_users), with real, specified attributes — gen_ai.provider.name, gen_ai.request.model, gen_ai.usage.input_tokens/output_tokens — and, critically, gen_ai.input.messages/gen_ai.output.messages as the spec's own designated place for the actual tool call arguments and results.
  • A real repro built exactly this: real spec-shaped spans around an agent call to a subtly buggy tool. The tool silently returned cumulative all-time users instead of yesterday's daily figure. The call completed with zero errors — stop_reason: "end_turn" — while the real final answer confidently claimed the number was "very healthy... roughly 46.8x the goal."
  • An independent, trace-only diagnosis — no access to the final answer, no re-running the agent — found the real root cause immediately: inspecting only the real gen_ai.output.messages attribute on the tool-execution span, the check correctly flagged the exact value as "implausibly large for a single day, likely a cumulative/all-time figure mislabeled as daily." This is the concrete, working version of the "3 a.m. question": can you tell what actually went wrong from the trace alone, at 3 a.m., without waking anyone else up to help re-run it?

Why "the call succeeded" is the wrong question for an agent

Traditional service monitoring answers one question well: did this request complete without an error? For a web server, that's usually the right question — a 200 status code is genuinely strong evidence the request worked. For an agent, it isn't. A tool call can return real, valid, well-formed JSON that's simply wrong; a model can reason fluently from that wrong data to a confident, articulate, incorrect conclusion; and every individual step along the way will report success, because nothing actually errored. The real, current framing for this gap: "When your AI agent returns a confidently wrong answer, your monitoring sees a successful 200 response." The same source's opening scenario makes the stakes concrete: "It is 3:07am on a Tuesday. Your AI agent...silently stops working. No alert fires" — not because nothing happened, but because what happened doesn't look like the kind of failure traditional monitoring is built to catch.

What OpenTelemetry's real GenAI conventions actually specify

A trace is a structured record of one step in an agent's run — what went in, what came out, how long it took, whether it succeeded — the same idea as a receipt for a single purchase, except the "purchase" here is a model call or a tool call. A span is one such entry inside a trace; a single agent turn usually produces several spans, one per step, so the full trace is just the sequence of receipts for everything that happened. Rather than every team inventing its own shape for these receipts, OpenTelemetry maintains real, versioned semantic conventions specifically for generative AI and agents — a standard format so a trace produced by one tool reads the same way as a trace from any other. The span-naming rule is concrete and mechanical: for tool execution, "Span name SHOULD be {gen_ai.operation.name} {gen_ai.tool.name}" — so a call to a tool named get_daily_active_users produces a span literally named execute_tool get_daily_active_users, not an opaque, team-specific label. The real, specified attributes for that span include gen_ai.operation.name (required; execute_tool is one of its defined values), gen_ai.tool.name (required), gen_ai.tool.call.id and gen_ai.provider.name (conditionally required), and token usage fields when available.

The detail that matters most for actual debugging — where the real tool call arguments and results get recorded — isn't a dedicated gen_ai.tool.call.arguments attribute; per the spec, that information belongs in gen_ai.input.messages and gen_ai.output.messages, structured JSON attached to the relevant messages, with an explicit, real privacy note attached: "This attribute is likely to contain sensitive information" and instrumentations "MAY provide a way for users to filter or truncate" it. Agent-level spans follow the same pattern one level up: invoke_agent {gen_ai.agent.name}, create_agent {gen_ai.agent.name}, plan {gen_ai.agent.name} — a consistent, standard shape for every layer of an agent's real execution.

Repro: a confidently wrong answer, and whether the trace alone can catch it

The code below defines the buggy tool itself — get_daily_active_users, which silently returns cumulative all-time users instead of yesterday's figure — the exact mislabeled value the run below has to be diagnosable from.

def get_daily_active_users(product: str) -> dict:
    """A real bug: this returns CUMULATIVE all-time users, not daily active users,
    despite the tool's name and description promising a daily figure. The bug is in
    the tool implementation, not in anything the model does."""
    _CUMULATIVE_USERS = {"Nimbus": 2_340_000}
    return {"product": product, "daily_active_users": _CUMULATIVE_USERS.get(product, 0)}


TOOLS = [{
    "name": "get_daily_active_users",
    "description": "Get yesterday's daily active user count for a product.",
    "input_schema": {"type": "object", "properties": {"product": {"type": "string"}}, "required": ["product"]},
}]

TASK = "What were yesterday's daily active users for 'Nimbus', and is that healthy against our 50,000 DAU target?"

The next block wires that tool into a real agent call and emits the trace using the actual OpenTelemetry GenAI attribute names, so what follows is a real spec-shaped trace, not a mocked one.

class Span:
    """Emits real OpenTelemetry GenAI semantic-convention attribute names, not a
    made-up schema -- gen_ai.operation.name, gen_ai.provider.name, gen_ai.tool.name,
    gen_ai.usage.*, gen_ai.input.messages / gen_ai.output.messages (the real spec's
    own place for tool call arguments and results), and error.type."""

    def __init__(self, name: str, attributes: dict):
        self.name = name
        self.attributes = attributes
        self.start = time.time()

    def to_dict(self) -> dict:
        return {"span_name": self.name, "duration_ms": round((time.time() - self.start) * 1000, 2), **self.attributes}


def run_traced(client, model: str, task: str) -> dict:
    spans = []
    messages = [{"role": "user", "content": task}]
    for _ in range(4):
        response = client.messages.create(model=model, max_tokens=600, tools=TOOLS, messages=messages)
        spans.append(Span(f"chat {model}", {
            "gen_ai.operation.name": "chat",
            "gen_ai.provider.name": "anthropic",
            "gen_ai.request.model": model,
            "gen_ai.response.model": response.model,
            "gen_ai.usage.input_tokens": response.usage.input_tokens,
            "gen_ai.usage.output_tokens": response.usage.output_tokens,
            "gen_ai.output.messages": [
                {"type": "text", "text": b.text} if b.type == "text"
                else {"type": "tool_call", "name": b.name, "arguments": b.input} if b.type == "tool_use"
                else {"type": b.type}
                for b in response.content
            ],
        }).to_dict())
        messages.append({"role": "assistant", "content": response.content})
        if response.stop_reason != "tool_use":
            break
        tool_results = []
        for block in response.content:
            if block.type != "tool_use":
                continue
            result = get_daily_active_users(**block.input)
            span = Span(f"execute_tool {block.name}", {
                "gen_ai.operation.name": "execute_tool",
                "gen_ai.tool.name": block.name,
                "gen_ai.input.messages": block.input,
                "gen_ai.output.messages": result,
            })
            spans.append(span.to_dict())
            tool_results.append({"type": "tool_result", "tool_use_id": block.id, "content": str(result)})
        messages.append({"role": "user", "content": tool_results})
    return {"spans": spans, "response": response}

A real run — a clean 'success,' a confidently wrong answer, and a real spec-shaped trace

Task: "What were yesterday's daily active users for 'Nimbus', and is that healthy against our 50,000 DAU target?" The real bug: get_daily_active_users actually returns cumulative all-time users, not yesterday's figure — a bug in the tool's own implementation, invisible to the model, which has no way to know the number it received doesn't mean what its name promises.

Traditional monitoring's real view of this run: {"status": "success", "stop_reason": "end_turn", "error": null} — nothing to flag, nothing wrong.

The real final answer: "Yesterday's DAU for Nimbus was 2,340,000. Compared to your target of 50,000 DAU, that's well above target — roughly 46.8x the goal (2,340,000 vs. 50,000). This is a very healthy result, so either Nimbus is significantly outperforming expectations, or it's worth double-checking that the 50,000 target is still current/accurate for this product's scale." Confident, well-reasoned, articulate — and wrong, because the underlying number was never a daily figure at all.

The real trace, using the actual spec's attribute names: a chat claude-sonnet-5 span (gen_ai.operation.name: "chat", gen_ai.provider.name: "anthropic", real token counts) followed by an execute_tool get_daily_active_users span carrying the real gen_ai.input.messages: {"product": "Nimbus"} and gen_ai.output.messages: {"product": "Nimbus", "daily_active_users": 2340000}.

def diagnose_from_trace_alone(spans: list) -> dict:
    """A real, independent diagnostic pass -- using ONLY the trace's own data,
    with no access to the agent's final answer and no re-running anything --
    checking whether the tool's returned number is plausible for a DAILY figure."""
    for span in spans:
        if span.get("gen_ai.operation.name") != "execute_tool":
            continue
        result = span.get("gen_ai.output.messages", {})
        dau = result.get("daily_active_users")
        if dau is not None and dau > 100_000:  # a daily figure this large for one product is implausible
            return {
                "diagnosable_from_trace_alone": True,
                "flagged_span": span["span_name"],
                "flagged_value": dau,
                "reason": f"get_daily_active_users returned {dau:,} -- implausibly large for a single day, likely a cumulative/all-time figure mislabeled as daily",
            }
    return {"diagnosable_from_trace_alone": False}

The independent, trace-only diagnosis — run with no access to the final answer, no re-running the agent, only the real trace's own data — inspected the execute_tool span's gen_ai.output.messages attribute and correctly flagged it: "get_daily_active_users returned 2,340,000 — implausibly large for a single day, likely a cumulative/all-time figure mislabeled as daily." The exact tool call, the exact wrong value, and a specific, correct hypothesis — found entirely from a trace built to the real, standard convention.

What this means in practice

The gap this page's repro makes concrete is the same gap the real "3 a.m." framing names directly: an agent's individual steps succeeding tells you almost nothing about whether the task succeeded, because the two things fail independently. A monitoring setup built around "did anything error" will miss this class of failure by construction — it's not a matter of better alerting thresholds, it's that the failure genuinely doesn't produce an error at all. What actually closes the gap is recording the real content of each step — not just that a tool ran, but what it was asked and what it returned — in a structured, standard shape a human (or a second, automated check) can inspect without reproducing the failure. That's the entire practical value of OpenTelemetry's GenAI conventions over an ad hoc logging scheme: gen_ai.input.messages and gen_ai.output.messages existing as real, specified, cross-vendor attribute names means any tool built against the convention can read a trace produced by any other, and the specific diagnostic check this page's repro ran — "is this value plausible for what it claims to be" — is exactly the kind of check that's only possible when the real inputs and outputs are actually captured, not summarized away.

Interview angle

Weak answer to "how would you monitor an agent in production?": "Set up alerts for errors and track uptime." This is the traditional-monitoring answer this page's own repro directly falsifies — the demo's agent call had zero errors and 100% uptime by that definition, while still producing a real, consequential wrong answer that traditional monitoring would never surface.

Strong answer: instrument every model call and tool call with structured, standard-shaped spans — specifically the real OpenTelemetry GenAI conventions (gen_ai.operation.name, gen_ai.tool.name, and the actual inputs/outputs via gen_ai.input.messages/gen_ai.output.messages) — so that a failure which doesn't produce an error can still be diagnosed from the trace's actual content. This page's own repro demonstrates the mechanism directly: a wrong answer with a clean success status was made diagnosable in one automated check, entirely from trace data, with no need to reproduce the failure live.

Follow-up to expect: "isn't logging every tool call's full input and output a privacy or cost problem?" Yes, and the real spec accounts for it directly — it explicitly warns that gen_ai.input.messages/gen_ai.output.messages are "likely to contain sensitive information" and states instrumentations "MAY provide a way for users to filter or truncate" them. The practical answer is the same discipline Context Editing and compaction techniques already require elsewhere in this project: capture what you need for real diagnosis, truncate or redact what you don't, and treat that boundary as a deliberate design decision rather than an afterthought — not "log everything forever" and not "log nothing to stay safe," either extreme defeats the purpose.

Build it yourself — 30 minutes

  1. Build one deliberately buggy tool whose output looks plausible but is subtly wrong (a units mismatch, a stale cache, a metric confused with a similarly-named one, the way this page's cumulative-vs-daily bug does). Run an agent against it and confirm the call completes with no errors.
  2. Wrap the call in real spans using the actual OpenTelemetry GenAI attribute names — gen_ai.operation.name, gen_ai.tool.name, gen_ai.input.messages, gen_ai.output.messages — rather than an ad hoc logging format. This page's own recipe is a minimal, working template.
  3. Write one independent diagnostic check that only reads the trace's own data — no access to the agent's final answer, no re-running anything — and see whether it can catch the real bug. This page's repro used a simple plausibility check (is this value too large to be what it claims); yours might check units, ranges, or consistency with another field.
  4. If your check catches the bug from the trace alone, that's the real, working version of the "3 a.m. question" — you now know what a genuinely useful trace needs to contain, instead of assuming your current logging already has it.

Scenario Check

Sources & further reading

  • OpenTelemetry, GenAI semantic conventions (semantic-conventions-genai repository) — the real span naming rules and gen_ai.* attribute definitions this page's repro is built against directly.
  • Jamie Mallers, "Your AI Agent Crashed at 3am and Nobody Noticed" (OneUptime, 2026-03-28) — the real "3 a.m." framing and the "confidently wrong answer... successful 200 response" quote this whole page is built around.
  • Evaluating Agents — outcome vs. trajectory grading, a closely related but distinct discipline: this page is about seeing what happened in production; that page is about grading whether it was right before deployment.
  • Benchmark Atlas — action-state grading (checking what actually happened, not what the reply claims), the same underlying discipline this page applies to live production traces instead of benchmark tasks.