Tracing and Observability for Agent Systems

An LLM call that fails badly and an LLM call that works perfectly look identical from outside the process. Julia Morgado, a community manager for OpenTelemetry, makes the point in one line in her PlatformCon talk: without instrumentation, a model call is just “an HTTP 200” (1:43). The status code is fine. The agent still looped four times, spent eleven thousand tokens, called the refund tool twice, and quietly recovered from a stale read.

That gap is the whole subject. Once a system can plan, call tools, retry, hand work to another agent, and pause for approval, the final answer stops being evidence about the run. You can get a correct outcome and still have no idea whether the run was efficient, stable, policy-safe, expensive, or rescued at the last moment by a downstream control.

This guide follows Evaluating Agent Trajectories, Not Just Outputs. That piece explains why the path matters. This one is about the evidence layer that makes the path visible, and it is more concrete than it could have been a year ago: there is now a named standard for agent telemetry, a set of span types to emit, and a handful of tools that read them.

TL;DR

Instrument agent runs as OpenTelemetry traces and follow the OpenTelemetry GenAI semantic conventions, which define invoke_agent, execute_tool, chat and related spans with gen_ai.* attributes. The agent and framework part of those conventions is status: Development — it can change in any release — so pin the semantic-conventions version you build against and expect attribute renames rather than treating the names as stable API.

Aim for one trace per user request, with spans for each model call, tool call, handoff, and approval wait. Keep prompt and response content capture opt-in and redacted, because the conventions make it opt-in for a reason. Then watch a small set of signals across runs — tool error rate, latency, cost, regenerations, plus classifier-derived signals like task failure and user frustration — and keep the classifiers binary.

Tracing, Monitoring, and Observability Are Not the Same Thing

These three get collapsed into one bucket, and the collapse causes real confusion when a team buys a trace viewer and expects an operating capability.

Tracing is the structured record of a specific run. Monitoring is the recurring watch layer over many runs. Observability is the broader capability that lets a team explain, diagnose, and improve what the system is doing from the evidence it emits.

If a customer asks why one refund run escalated late, you need tracing. If retry rates doubled across the last thousand runs, you need monitoring. If your team wants to connect those failures back to tool changes, policy checks, model versions, and review workflows, you are doing observability.

What a Trace Actually Is

A useful agent trace is a structured, connected record of a run: the request that started it, the steps it moved through, the tool calls it made, the observations it got back, the retries and pauses and handoffs it took, the approvals and guardrails it hit, and the outcome it reached.

Plain logs tell you that events happened. Traces tell you how those events relate. In practice there are four objects worth keeping distinct:

• Trace — the full record of one coherent task execution. For agents that usually means one user request, one orchestration path, one coding task, one support case.
• Span — one bounded unit inside that trace: a model call, a tool invocation, a planning step, a subagent handoff, an approval pause, a retrieval. This is the level at which tracing becomes operationally useful, because you can see how the run decomposed, how long each part took, and where the path changed.
• Event — a meaningful occurrence inside a span: a retry, a timeout, a guardrail hit, a tool error, a human approval or rejection. This matters because many agent failures are not clean span-level failures. The span completes. The signal is that something unusual happened inside it.
• Thread or conversation correlation — the id that links related runs across turns, sessions, reopened tasks, and chained workflows. Without it you can understand a run locally and still miss the pattern the user actually experienced over a week.

The target shape is worth stating plainly, because it is the design decision that most often gets made by accident. The KubeCon deep dive on agent observability puts it as “a single unified trace per user request paints… the big picture” (19:28). One request, one trace, everything else nested underneath.

The same talk names why agent traces are harder than the service traces most teams already run: the agentic stack carries fluid state where traditional systems carry explicit state (12:40). A microservice’s state is in a database row you can query after the fact. An agent’s state is partly in a context window that no longer exists. If the span did not record it, it is gone.

The Standard: OpenTelemetry GenAI Semantic Conventions

This is no longer a loose best-practices topic, and the specific thing to name is the OpenTelemetry GenAI semantic conventions. They define what a GenAI span is called and which attributes it carries, so that a trace emitted by your code means the same thing to Langfuse, Phoenix, or a Grafana stack. The KubeCon deep dive is given by a member of the OpenTelemetry GenAI SIG — the group “responsible for coming up with semantic conventions for genai based apps” (1:05) — and it is candid about the state of play: agentic observability conventions, along with vector database conventions, are “work in progress” (21:02).

That is the important caveat and it is written into the spec itself. The agent and framework spans document carries Status: Development, as does the inference span definition. In OpenTelemetry’s maturity ladder, Development means the shape can change in any release, without the stability guarantees that apply to a stable convention. Two further signs of a moving target: the GenAI conventions now live in their own repository (semantic-conventions-genai) rather than inside the main semantic-conventions repo, and the Python package opentelemetry-semantic-conventions was at 0.66b1 (released 2026-10-06) — still a beta version number.

None of that is a reason to avoid the conventions. It is a reason to treat the attribute names as versioned data rather than a stable API:

• Pin the semantic-conventions version you build against, and bump it deliberately.
• Expect renames. Budget for a mechanical find-and-replace pass when you upgrade, and keep the attribute names in one module rather than scattered through call sites.
• Do not block on stability. The span structure — one trace per request, agent spans wrapping model and tool spans — is far more stable than the attribute spellings, and it is the part that carries most of the debugging value.

The span types

The conventions define a specific vocabulary for agent systems. The agent and framework spans document covers: create agent, invoke agent (both a remote client variant and an in-process internal variant), invoke workflow, plan, execute tool, load skill, read skill resource, and command execution.

The naming rules matter, because consistency is what makes traces queryable:

• gen_ai.operation.name holds the operation, with values including create_agent, invoke_agent, invoke_workflow, execute_tool, and chat.
• An agent invocation span is named invoke_agent {gen_ai.agent.name} — so invoke_agent support-bot, not a free-text description.
• Span kind follows the call boundary: a remote agent invocation is CLIENT, an in-process one is INTERNAL.
• An inference span is named {gen_ai.operation.name} {gen_ai.request.model} — chat gpt-4.1, for instance.
• Span status follows OpenTelemetry’s general Recording Errors guidance, which means failures set error.type rather than inventing a per-framework error field.

That last point is the one people skip, and it is the one that makes error rates comparable across a system where three teams instrumented three different frameworks.

One Request, One Trace

Here is what the shape looks like for a single support request that ends in a refund behind a human approval.

Emitting the Spans Yourself

The fastest way to understand the conventions is to emit them by hand once, with no vendor involved. This script produces a real trace with an invoke_agent root and nested chat and execute_tool spans, printed to your console.

# trace_agent_run.py — one agent run as an OTel trace, printed to stdout.
# pip install opentelemetry-sdk
# Run: python trace_agent_run.py
from __future__ import annotations

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor
from opentelemetry.trace import SpanKind, Status, StatusCode

# The GenAI conventions are Development status, so the attribute names live in
# one place and get re-checked whenever the pinned semconv version moves.
OPERATION = "gen_ai.operation.name"
AGENT_NAME = "gen_ai.agent.name"
CONVERSATION = "gen_ai.conversation.id"
PROVIDER = "gen_ai.provider.name"
REQUEST_MODEL = "gen_ai.request.model"
INPUT_TOKENS = "gen_ai.usage.input_tokens"
OUTPUT_TOKENS = "gen_ai.usage.output_tokens"
TOOL_NAME = "gen_ai.tool.name"
TOOL_CALL_ID = "gen_ai.tool.call.id"
ERROR_TYPE = "error.type"

# Content capture is opt-in in the conventions. Leave it off until redaction
# exists, and never flip it on in production as a debugging shortcut.
CAPTURE_CONTENT = False

provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("support-bot")


def chat(model: str, prompt: str, *, reply: str, tokens: tuple[int, int]) -> str:
    """One model call. Span name is '{operation} {request model}' per semconv."""
    with tracer.start_as_current_span(f"chat {model}", kind=SpanKind.CLIENT) as span:
        span.set_attribute(OPERATION, "chat")
        span.set_attribute(PROVIDER, "openai")
        span.set_attribute(REQUEST_MODEL, model)
        span.set_attribute(INPUT_TOKENS, tokens[0])
        span.set_attribute(OUTPUT_TOKENS, tokens[1])
        if CAPTURE_CONTENT:
            span.set_attribute("gen_ai.input.messages", prompt)
            span.set_attribute("gen_ai.output.messages", reply)
        return reply


def execute_tool(
    name: str, call_id: str, *, needs_approval: bool = False, approved: bool = True
) -> str:
    """One tool call. A refused approval is an error on the span, not silence."""
    with tracer.start_as_current_span(f"execute_tool {name}", kind=SpanKind.INTERNAL) as span:
        span.set_attribute(OPERATION, "execute_tool")
        span.set_attribute(TOOL_NAME, name)
        span.set_attribute(TOOL_CALL_ID, call_id)
        if needs_approval:
            # The wait stays inside the span so the delay shows up as latency.
            span.add_event("human_approval.requested", {TOOL_NAME: name})
            if not approved:
                span.add_event("human_approval.rejected", {TOOL_NAME: name})
                span.set_attribute(ERROR_TYPE, "approval_rejected")
                span.set_status(Status(StatusCode.ERROR, "rejected by reviewer"))
                return "rejected"
            span.add_event("human_approval.granted", {TOOL_NAME: name})
        return "ok"


def handle_request(conversation_id: str) -> str:
    """The root span: 'invoke_agent {agent name}', INTERNAL because it is in-process."""
    with tracer.start_as_current_span("invoke_agent support-bot", kind=SpanKind.INTERNAL) as span:
        span.set_attribute(OPERATION, "invoke_agent")
        span.set_attribute(AGENT_NAME, "support-bot")
        span.set_attribute(CONVERSATION, conversation_id)

        chat("gpt-4.1", "refund request", reply="search_orders", tokens=(820, 40))
        execute_tool("search_orders", "call_01")
        chat("gpt-4.1", "two orders found", reply="refund order 8812", tokens=(1340, 96))
        result = execute_tool("refund", "call_02", needs_approval=True, approved=True)

        if result != "ok":
            span.set_attribute(ERROR_TYPE, "approval_rejected")
            span.set_status(Status(StatusCode.ERROR, "refund not completed"))
        return result


if __name__ == "__main__":
    print(handle_request("conv_7731"))
    provider.shutdown()  # Flush before the process exits.

Run it and you get five spans sharing one trace id, with the agent span as the parent. That is the whole trick: the nesting is what turns a pile of model calls into a trajectory you can read.

Switching to a backend is a three-line change. Any OTLP endpoint works, including Langfuse and Phoenix:

# pip install opentelemetry-exporter-otlp-proto-http
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))

OTLPSpanExporter reads OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS from the environment, so the vendor choice becomes config rather than code. Use BatchSpanProcessor rather than SimpleSpanProcessor anywhere real; the simple one exports synchronously on span end, which is what you want for a console demo and not what you want on a request path.

Who wires this up

Doing the above per service, per team, is how instrumentation rots. Morgado’s PlatformCon talk is pointed about it: hand-wiring SDKs team by team is brittle (3:15), and the alternative is a platform-level approach where instrumentation is injected rather than written into every application (3:46). Her demonstration uses a Kubernetes operator that is “AI framework aware” — the OpenLIT operator (4:47) — so applications get GenAI instrumentation without each team adding SDK calls.

The cost argument is the one that usually wins the budget conversation. Without per-request token counts on the span, you cannot attribute spend to a feature, a customer, or a prompt change (2:13) — you get one large monthly invoice and a guess.

The Tool Landscape

You do not have to pick a vendor to start, which is the point of the console example above. But most teams end up with a backend, and the useful axes are: does it speak OpenTelemetry natively, can you run it yourself, and does it do evaluation as well as tracing?

Two notes on reading that table. First, deployment and pricing options move faster than any article, so each row links to the product’s own docs, which are the authority. Second, the vendor-native row is the one with the sharpest tradeoff: the OpenAI Agents SDK gives you tracing with zero setup, and its exporter-processor hook is the seam you use to get that data into an OTel pipeline. (The KubeCon talk describes that SDK as shipping with standard OpenTelemetry semantic conventions, at 21:02. The SDK’s own docs describe a built-in tracing model with exporter processors rather than semconv-native output, so plan for a bridge and verify against the docs for the version you install.)

For versions as of this writing: langfuse 4.17.0 (2026-10-05) and arize-phoenix 20.21.1 (2026-10-12). Both move quickly. Pin them.

If you are choosing a backend mainly to read traces by hand, the annotation surfaces matter more than the ingest path, and Error Analysis: The First Eval You Should Run covers that comparison from the reviewer’s side.

The Signals Worth Watching

A trace explains one run. Monitoring is about what you watch across thousands, and the most useful framing here comes from the Raindrop team’s AI Engineer talk on agent observability, which splits agent signals into two kinds.

Explicit signals are the ones the system already emits: tool error rate, latency, users regenerating a response, and cost (3:52). These are cheap, unambiguous, and should be on a dashboard before anything fancier. Note that “users regenerating” is a product signal rather than an infrastructure one — it is a user telling you the answer was wrong, in a form you can count.

Implicit signals are the ones you have to derive from the content of the run: regex signals, classifiers, and self-diagnostics (4:22). The ones they see most often are task failure, user frustration, content moderation, jailbreaking, and — worth noting, because nobody instruments for it — wins (5:23).

One design rule from the same talk is worth adopting directly: prefer binary issue classifiers over 1-to-10 ratings (4:53). A classifier that answers “did this run frustrate the user, yes or no” produces a number you can alert on and compare week over week. A 1-to-10 frustration score produces a number nobody can calibrate, including the model that emitted it. This is the same conclusion error analysis arrives at from the annotation side, and it is not a coincidence — both are fighting the same drift in what a middling score means.

What to do with the signals: alerting and experiments (7:26). That second use is the one that closes the loop, and it is where this topic meets Online Evals vs Offline Evals — signals computed over live traces are your online eval layer, and the traces they run on are the dataset your offline suite gets built from. (Raindrop sells an agent observability product, so take the framing as a practitioner’s view of what they see across customers rather than a neutral survey.)

There is one more signal worth attaching that is easy to miss: user identity. The KubeCon talk recommends putting user ids on spans so traces can be aggregated per user and the result fed back into memory (16:50). That is how you answer “is this one angry customer or a pattern” without guessing — though see the next section, because user ids are exactly the field that needs a retention policy.

Rounding out the agent-specific set beyond the generic infrastructure metrics: per-step latency, tool success rates, retry frequency, cost per successful task, escalation rates, guardrail-hit rates, and abandonment or timeout patterns. Cost per successful task is the one most worth adding early, because an agent that succeeds on the third attempt is cheap-looking in a per-call metric and expensive in reality.

Capture Enough Evidence, Not Every Possible Byte

Once teams understand traces matter, the reflex is to capture everything. That is the wrong target, and the conventions themselves make the point: capturing prompt and response content is opt-in, not default. Agent runs touch user messages, retrieved documents, tool arguments, internal identifiers, and policy decisions. A trace store with content capture switched on globally is a copy of your most sensitive data in a system that was provisioned for debugging.

The workable position is structure by default, content by exception:

• Always capture the shape of the run — spans, timings, tool names, token counts, error types, approval events. This is most of the debugging value and almost none of the risk.
• Capture content selectively, behind a flag, with redaction applied before export rather than after storage.
• Give content a shorter retention than structure. Thirty days of prompts and a year of span metadata is a defensible combination.
• Decide who can read which layer, and remember that a trace viewer that shows raw prompts is a data-access tool wearing an engineering badge.

The question is not “how do we record everything” but “how do we preserve enough evidence to debug, evaluate, and govern the system without turning the trace layer into uncontrolled data exhaust.” Weak capture leaves the system opaque; overbroad capture creates a security problem that outlives the bug you were chasing.

A Minimum Useful Evidence Set

If you want a checklist rather than a standard, the four things a run needs to emit are context, actions, signals, and enforcement.

Context is identity and state: run id, task or session id, agent name, model and configuration version, environment or tenant, and whatever state marker the agent believed it was operating under. In semconv terms this is mostly gen_ai.agent.name, gen_ai.conversation.id, and gen_ai.request.model.

Actions are what the system did: plans and subtasks, tool calls and arguments, tool results, handoffs, and approval requests. These are your execute_tool and invoke_agent spans.

Signals are the performance and failure surface: per-step latency, tokens, cost, retries, errors, timeouts, status changes. Spans give you latency for free; the rest are attributes you have to set.

Enforcement is the control layer: guardrail checks, permission checks, execution-boundary decisions, approval gates, policy assertions, evaluation hooks. This is the layer most often missing, and its absence is specifically expensive — a run that looks successful may have been rescued by a boundary that blocked an unsafe action. A blocked unsafe action is not proof of a good run. It is proof that a control caught a bad one, and you want that on the record. This is where tracing meets Human-in-the-Loop Control Design: if a human approves, rejects, or overrides a step, that decision belongs in the trace.

Failure Modes When Teams Skip This Layer

Silent recovery hides brittleness. The run reaches a correct answer after failed tool calls, stale reads, and retries. The team sees success. The trace would have shown a system one timeout away from a bad day. This is the main mechanism behind The Most Common Ways Agents Fail Silently.

Policy incidents become unauditable. If a boundary blocked something unsafe and nothing recorded it, the review after the next incident has no history to read.

Multi-agent systems become uninspectable. Once work moves between supervisors, routers, and executors, transcripts stop being enough — you need trace structure to see where control went and why. See Supervisor, Router, and Planner-Executor Patterns.

Evaluation loses its evidence base. Without the steps, tools, retries, and checkpoints, trajectory evaluation has nothing to score, and teams end up grading polished outputs while the expensive behavior underneath goes unmeasured. The reverse is also true, and it is the better reason to instrument early: traces are the raw material for Traces as Test Data, and drift only becomes visible when you have a history to compare against.

A Practical Starting Point

You do not need a platform on day one. You need the run to emit enough structure that it can be reconstructed later.

1. One trace per user request. Get the nesting right before you get the attributes right.
2. Spans for every model call, tool call, handoff, and approval wait, named per the conventions so they stay queryable.
3. A conversation or session id on the root span, so multi-turn behavior is correlatable.
4. Token counts and error types on every span, because cost and failure attribution are the two questions you will be asked first.
5. Structure always, content behind a flag with redaction.
6. A small set of explicit signals on a dashboard — tool error rate, latency, cost per successful task, regenerations — before any classifier work.
7. A pinned semconv version recorded somewhere a future upgrade will find it.

That is enough to debug runs, review controls, and build trajectory evaluation on evidence instead of memory. From there it grows into the broader operating discipline covered in AgentOps: Running Agents in Production.

Tracing is how you reconstruct a run. Monitoring is how you watch behavior across runs. Observability is how those become an operating capability rather than an archive. Without all three, you are judging a moving system from its ending — or from an HTTP 200.

FAQ

Is tracing the same as observability?

No. Tracing is the structured record of a run. Observability is the broader ability to explain and improve system behavior using traces, metrics, events, and the workflows built around them. A trace viewer is a component of observability, not a synonym for it.

Should I wait for the GenAI conventions to stabilize?

No. Build on them now, pin the version, and keep the attribute keys in one module. The span structure — one trace per request, agent spans wrapping model and tool spans — is the durable part and carries most of the value. The Development status means names can change between releases, which is an upgrade cost, not a reason to emit unstructured logs in the meantime.

Do I need a vendor to do this?

No. The console example above is a complete trace with no backend at all, and switching to any OTLP endpoint later is a three-line change plus environment variables. Start with the OpenTelemetry SDK and pick a backend when reading traces by hand becomes the bottleneck.

Is this just logging with extra steps?

No. Logs are isolated event messages. Traces add the parent-child relationships between steps, model calls, and tool calls, which is precisely what you need when the question is “why did it call refund twice” rather than “did an error occur.”

How is this different from trajectory evaluation?

Tracing captures the evidence of what happened. Trajectory evaluation judges whether what happened was good. Evaluation depends on tracing, but they are different jobs, and conflating them is how teams end up with a scoring pipeline running on data too thin to score.

Do small agents need tracing?

Not a full stack. But as soon as a system takes multiple steps, calls tools, creates side effects, or crosses an approval boundary, structured traces start paying for themselves within the first debugging session. Capturing span structure without content is cheap enough that there is rarely a good reason to defer it.

Should I put user ids on spans?

It is genuinely useful — it is how you aggregate per user and tell a pattern from a one-off. Treat it as personal data: use a stable pseudonymous id rather than an email where you can, and give it a retention policy on day one rather than after the first access review.

Further watching

Each of these was caption-checked for this guide; timestamps link to the moment the claim appears.

• Deep Dive To AI Agent Observability — Guangya Liu (IBM) & Karthik Kalyanaraman (Langtrace) (CNCF / KubeCon EU 2025, 31 min). The standards view from inside the OpenTelemetry GenAI SIG. Implicit versus explicit state at 12:40, per-user trace aggregation at 16:50, the one-trace-per-request argument at 19:28, and the frank “work in progress” status at 21:02.
• Everything You Need To Know About Agent Observability — Danny Gollapalli & Zubin Koticha (Raindrop) (AI Engineer, 2026, 50 min). The signals taxonomy this guide uses. Explicit signals at 3:52, implicit signals at 4:22, the case for binary classifiers at 4:53, and what to do with the signals at 7:26. A vendor talk; read it as a practitioner’s field view.
• Making AI workloads observable by default on Kubernetes — Julia Morgado (PlatformCon 2026, 11 min). The shortest watch here and the best one on who instruments. The HTTP 200 framing at 1:43, per-request token cost at 2:13, why per-team SDK wiring is brittle at 3:15, and the operator approach at 3:46.

Primary Sources

• OpenTelemetry GenAI semantic conventions — the conventions surface, including the inference span naming rule and its Development status.
• GenAI agent and framework spans — Status: Development; the span types (create agent, invoke agent, invoke workflow, plan, execute tool, load skill, read skill resource, command execution), the gen_ai.operation.name values, the invoke_agent {gen_ai.agent.name} naming rule, the CLIENT and INTERNAL span-kind split, and error recording via error.type.
• OpenAI Agents SDK tracing — built-in traces and spans, and the exporter processors used to forward them.
• Langfuse observability docs — the OpenTelemetry-based ingest path. langfuse 4.17.0 (2026-10-05).
• LangSmith observability docs — runs, threads, and the tracing model.
• Arize Phoenix docs — OpenInference conventions on OpenTelemetry. arize-phoenix 20.21.1 (2026-10-12).
• Braintrust docs — tracing alongside evaluation.
• opentelemetry-semantic-conventions 0.66b1 (2026-10-06) — the Python package carrying the convention constants, still on a beta version number.

Last verified: Oct 2026 (2026-10-13). OpenTelemetry GenAI semantic conventions status: Development. Package versions and URLs checked on that date.