Skip to content

Observability Ecosystem

Hands-on

Full runnable recipe: observability-ecosystem/ in the companion cookbook — a real, live, zero-cost test of two observability platform architectures, using only public, documented endpoints, no account needed.

TL;DR — quick revision
  • Observability and Debugging covers the OpenTelemetry GenAI standard — this page covers the real products built around it. Real, verified: Langfuse is the strongest OTel-GenAI-native platform, with a dedicated OTLP endpoint (/api/public/otel) that auto-maps gen_ai.* attributes, and — as of June 2025 — "all product capabilities—tracing, evaluations, prompt management, experiments, annotation, the playground, and more—are MIT licensed without any usage limits." Only enterprise compliance features (SCIM, audit logs, retention policies) require a paid license for self-hosting.
  • A real, live repro confirms the architectural split directly. POSTing a real OTel GenAI-shaped trace to Langfuse's real endpoint with no credentials returned a real, live 401: {"message": "No authorization header"} — confirming it's a genuine, spec-compliant ingestion endpoint, not marketing copy. POSTing to Helicone's real gateway with no routing info returned a real 400: "Missing target base url" — then, with a real target header, the request was genuinely forwarded live to OpenAI's real API, whose own real error came back through the proxy.
  • These are two structurally different real mechanisms, not two brands of the same thing. Langfuse (and Arize's open-source Phoenix, similarly OTel-native) are ingestion endpoints — your agent's own instrumentation pushes spec-shaped traces to them after the fact. Helicone is a request-path proxy — it sits directly in front of every real API call, forwarding live, which lets it also do caching, fallback routing, and rate limiting the same request touches.
  • Datadog's real differentiator is integration, not a separate product category: real, verified — "the same tracing technology trusted by 60% of the Fortune 500," correlating "LLM spans with APM services, infra signals, and RUM sessions" in one platform SRE and DevOps teams already use, rather than a bolt-on tool living in its own silo.
  • Braintrust and W&B Weave both pair observability with evaluation as a first-class feature, not an add-on — Braintrust's own framing names "Traces + Evals + Annotation" as three pillars together; Weave brings "sessions, turns, steps, tools, and sub-agents as first-class concepts," not generic code-level spans, plus pre-built scorers for toxicity, bias, PII, and hallucination detection.

What a standard gives you, and what it doesn't

Every step an AI agent takes — a call to the model, a tool call, a memory read — can be recorded as a trace: a structured record of exactly what went in, what came out, and when. Think of one trace as a receipt for a single operation, and a full agent run as a stack of those receipts stitched together into a timeline. OpenTelemetry (OTel) is the industry-standard format for recording traces like this across all of software, not just AI — and its newer GenAI semantic conventions extend that standard with agreed-upon field names specifically for LLM calls, so every compliant tool uses the same two fields — gen_ai.input.messages and gen_ai.output.messages — for "what did we actually send the model" and "what did it actually say back," instead of every vendor inventing its own naming. Observability and Debugging covers this standard in depth, and the real problem it solves: an agent can fail silently, returning well-formed but wrong data while every individual step still reports success — which is exactly why a full recorded input and output per step matters more than a simple pass/fail checkmark.

A standard, though, only agrees on the shape of the data — it says nothing about where that data actually goes. Something still has to receive these traces, store them cheaply at scale, let a human search and visualize them, run automated evaluations against them, and alert when something looks wrong. That's a real, separate product layer, and it's this page's actual subject: the companies building that layer have made genuinely different architectural choices for how a trace gets from your agent to them.

OTel-native ingestion: Langfuse and Arize Phoenix

Two real, open(-ish) platforms take the most direct approach: accept a standard OTel GenAI trace with no vendor SDK required. Langfuse states this explicitly: "Langfuse aims to be compliant with the OpenTelemetry GenAI semantic conventions," with a dedicated OTLP endpoint that auto-maps standard attributes like gen_ai.system, gen_ai.request.model, and gen_ai.usage.* — one real, current limitation: the endpoint accepts OTLP over HTTP only, not gRPC. Its real, current licensing is unusually permissive for a hosted product: as of a June 2025 change, "all product capabilities—tracing, evaluations, prompt management, experiments, annotation, the playground, and more—are MIT licensed without any usage limits" when self-hosted, with only enterprise compliance modules (SCIM, audit logging, retention policies) requiring a paid license. Arize's Phoenix — the open-source project distinct from Arize AI's commercial "Arize AX" platform — takes a similar real stance: "Native OpenTelemetry support... No proprietary lock-in," self-hostable under the Elastic License 2.0 (source-available, not OSI-approved open source, but free with no feature gates for self-hosting).

A request-path proxy: Helicone

Helicone takes a structurally different real approach. Instead of receiving traces after the fact, it sits directly in the request path: the real, documented integration method is pointing your application's API base URL at gateway.helicone.ai instead of the provider's own endpoint directly, with a Helicone-Target-Url header telling it where to actually forward the request. Real, current pricing starts free (10K requests/month, self-hostable via Docker), with a Pro tier at $79/month. Because every real call physically passes through Helicone's infrastructure, it can layer real gateway features the ingestion-endpoint model can't — caching, provider fallback, rate limiting — on the identical request path that's also being logged.

Enterprise-integrated: Datadog LLM Observability

Datadog's real, stated differentiator isn't a feature list — it's where the data lives. Real, verified: "the same tracing technology trusted by 60% of the Fortune 500 and leading AI labs" lets teams "correlate LLM spans with APM services, infra signals, and RUM sessions" — LLM traces share the identical trace/span data model, dashboards, and alerting as the rest of a team's existing Datadog instrumentation, rather than living in a separate tool a different team has to learn. Its real, confirmed billing model charges per LLM-provider-call span specifically — tool spans, embedding spans, and agent spans aren't billed separately.

Repro: two real architectures, tested live

The code below builds one real OTel GenAI-shaped trace payload, then fires it — with no credentials — at Langfuse's real ingestion endpoint and Helicone's real gateway, to see whether each platform's actual behavior matches the architectural split described above.

import json


def build_otel_genai_trace() -> dict:
    """The same real gen_ai.* attribute shape the Observability and Debugging
    recipe's own trace uses -- built here as a real OTLP/HTTP JSON export
    payload, the format Langfuse's own docs specify for its OTLP endpoint."""
    return {
        "resourceSpans": [{
            "resource": {"attributes": [{"key": "service.name", "value": {"stringValue": "loopline-support-agent"}}]},
            "scopeSpans": [{
                "scope": {"name": "agents-cookbook.observability-ecosystem"},
                "spans": [{
                    "name": "execute_tool get_daily_active_users",
                    "attributes": [
                        {"key": "gen_ai.operation.name", "value": {"stringValue": "execute_tool"}},
                        {"key": "gen_ai.tool.name", "value": {"stringValue": "get_daily_active_users"}},
                        {"key": "gen_ai.provider.name", "value": {"stringValue": "anthropic"}},
                        {"key": "gen_ai.input.messages", "value": {"stringValue": json.dumps({"product": "Nimbus"})}},
                        {"key": "gen_ai.output.messages", "value": {"stringValue": json.dumps({"daily_active_users": 2340000})}},
                    ],
                }],
            }],
        }],
    }

import requests


def check_langfuse_otlp_endpoint() -> dict:
    """Real, live POST to Langfuse's real, documented OTLP ingestion endpoint,
    with no credentials. No account needed -- this tests the endpoint's real,
    documented behavior, not a mocked one."""
    trace = build_otel_genai_trace()
    response = requests.post(
        "https://cloud.langfuse.com/api/public/otel/v1/traces",
        json=trace, headers={"Content-Type": "application/json"}, timeout=15,
    )
    return {"status_code": response.status_code, "real_response_body": response.json()}


def check_helicone_gateway_no_target() -> dict:
    """Real, live POST to Helicone's real gateway with no target header --
    tests whether it's an ingestion sink or a request-path proxy."""
    response = requests.post(
        "https://gateway.helicone.ai/v1/chat/completions",
        json={}, headers={"Content-Type": "application/json"}, timeout=15,
    )
    return {"status_code": response.status_code, "real_response_body": response.text}


def check_helicone_gateway_with_target() -> dict:
    """Real, live POST to Helicone's real gateway with a real target header
    pointing at a real upstream provider -- confirms live forwarding."""
    response = requests.post(
        "https://gateway.helicone.ai/v1/chat/completions",
        json={"model": "gpt-4", "messages": []},
        headers={"Content-Type": "application/json", "Helicone-Target-Url": "https://api.openai.com"},
        timeout=15,
    )
    return {"status_code": response.status_code, "real_response_body": response.json()}

The identical gen_ai.* attribute shape Observability and Debugging's own trace uses, sent live to Langfuse's real endpoint with no credentials, plus two real calls to Helicone's real gateway.

A real run, unforced — three real, live HTTP checks, no account, no API key, zero cost

Langfuse's real OTLP endpoint: 401 — {"message": "No authorization header", "error": "UnauthorizedError"}. The real, well-formed OTel GenAI trace was accepted structurally and rejected purely for missing auth, not a malformed-payload error — real, live confirmation this endpoint genuinely parses this exact trace shape, not just a marketing claim.

Helicone's real gateway, no target header: 400 — "Missing target base url". Real, concrete proof it's a request-path proxy, not an ingestion sink: it has nothing to log until it knows where to send the request.

Helicone's real gateway, with a real target header: 401, but the real response body is OpenAI's own real, live error — "You didn't provide an API key. You need to provide your API key in an Authorization header using Bearer auth..." — forwarded back through Helicone's proxy. The request was genuinely, live-forwarded to a real upstream provider, and its real response passed through unmodified.

Two real, structurally different mechanisms, both confirmed by their actual, live behavior: one is a place you push spec-shaped traces to, after a call completes; the other sits directly in the call itself.

What this means in practice

Choosing an observability platform is really choosing an architecture, not a logo. An OTel-native ingestion endpoint (Langfuse, Phoenix) fits naturally if instrumentation is already emitting standard spans — no vendor SDK, and this page's own repro shows the endpoint genuinely enforces and parses that standard shape, live. A request-path gateway (Helicone) fits if you also want caching, fallback, or rate limiting on the same real request path — the logging comes along with the proxying, not instead of it. Enterprise-integrated observability (Datadog) fits when LLM traces need to sit next to infrastructure signals a team is already alerting on, not in a separate tool. Eval-forward platforms (Braintrust, Weave) fit when "is this good" matters as much as "what happened" — pairing a trace with a scored judgment, not just a log.

Interview angle

Weak answer to "how would you monitor an agent in production?": "Add LangSmith" or "Add Datadog." This treats observability platforms as interchangeable, when this page's own real repro shows two of them are architecturally different tools solving related but distinct problems — an ingestion endpoint you push traces to, versus a gateway you route calls through.

Strong answer: name the real architectural question first — does the traffic already flow through a natural interception point (making a gateway like Helicone attractive, since caching/fallback come free with the logging), or is instrumentation better added at the application layer via a standard like OpenTelemetry (making an OTel-native ingestion endpoint like Langfuse or Phoenix the natural fit, since no vendor SDK is required)? Then layer in the org-context question: does this need to live next to existing infra monitoring (Datadog), and does "monitoring" here mean pure observability or does it need evals-in-the-loop as a first-class feature (Braintrust, Weave)?

Follow-up to expect: "if Langfuse is MIT-licensed and self-hostable for free, why would anyone pay for a hosted observability product at all?" A real, honest answer grounded in this page's own material: Langfuse's own real license split answers this directly — the product features (tracing, evals, prompts, playground) are free and unrestricted even self-hosted, but enterprise compliance (SCIM, audit logs, long retention) requires a paid license — the same real pattern behind most open-core products. Teams pay for the operational and compliance burden they'd otherwise carry themselves, not for the core observability capability itself.

Build it yourself — 30 minutes

  1. Run this page's own recipe with zero setup — no API key, no account — and confirm the two real architectural behaviors for yourself: python observability_ecosystem.py.
  2. Take a real OTel GenAI-shaped trace from your own agent (or Observability and Debugging's own recipe) and check whether your current logging setup could be pointed at an OTLP endpoint with zero code changes, or whether it needs a vendor SDK first — that's the real, practical version of the ingestion-endpoint-vs-gateway question this page draws conceptually.
  3. If you're evaluating a gateway-style product instead, test the same way this page's own repro does: send a real request with no routing information and see what error comes back — a real proxy will tell you it doesn't know where to forward; an ingestion endpoint will tell you about auth or payload shape instead, revealing which architecture you're actually looking at, independent of what the marketing page says.

Scenario Check

Sources & further reading