BuildPython

OpenTelemetry GenAI — Tracing Tool Calls End-to-End

An agent calls five tools, three MCP servers, and two sub-agents, and you need one trace across all of it. The OpenTelemetry GenAI semantic conventions (stable attributes in v1.37 and up) are the 2026 standard, natively supported by Datadog, Langfuse, Arize Phoenix, OpenLLMetry, and AgentOps. This lesson names the required attributes, walks the span hierarchy (agent → LLM → tool), and ships a stdlib span emitter you can plug into any OTel exporter.

TIME: ~75 minutesPREREQUISITES: Phase 13 · 07 (MCP server), Phase 13 · 08 (MCP client)

Learning Objectives

Name the required OTel GenAI attributes for an LLM span and a tool-execution span.
Build a trace hierarchy that covers agent loop, LLM call, tool call, and MCP client dispatch.
Decide what content to capture (opt-in) vs redact (defaults).
Emit spans to a local collector (Jaeger, Langfuse) without rewriting tool code.

The Problem

A debug from February 2026: a user reports "my agent sometimes takes 30 seconds to respond; other times 3 seconds." There are no traces. Logs show the LLM call, but not the tool dispatch, the MCP server round-trip, or the sub-agent. So you guess, and eventually you find that one MCP server occasionally hangs on a cold start.

Without end-to-end tracing you cannot find that. OTel GenAI is how you get it.

The conventions settled in 2025-2026 under the OpenTelemetry semantic-conventions group. They define stable attribute names, so Datadog, Langfuse, Phoenix, OpenLLMetry, and AgentOps parse the same spans. You instrument once and ship to whichever backend you pick.

Warm-up0/2

1. An MCP client launches a server as a child process and talks to it over stdio. What is true of that transport?Warm-up

Two boxes. An MCP client is the parent process and an MCP server is its child process. Two arrows labelled pipe 1 and pipe 2 run between them, one in each direction. — Schematic of the setup in the stem, typed.

2. How does an MCP client learn which tools a connected server offers?Warm-up

Schematic of the setup in the stem, typed.

Check: The Problem0/2

1. What do the GenAI semantic conventions define?Define

Two cards listing attribute names and values from a run of main.py: an llm.chat span with operation name chat, provider openai and request model gpt-4o, and a tool.execute span with operation name execute_tool. — Excerpts of the spans main.py builds; other attributes omitted.

2. Why were logs alone not enough to explain the agent that took 3 seconds sometimes and 30 seconds other times?Explain

Two horizontal bars drawn to scale for the same agent: run A lasts 3 seconds and run B lasts 30 seconds. — Durations from the stem; bars drawn to scale. Typed.

The Concept

Span hierarchy

agent.invoke_agent  (top, INTERNAL span)
 ├── llm.chat       (CLIENT span)
 ├── tool.execute   (INTERNAL)
 │    └── mcp.call  (CLIENT span)
 ├── llm.chat       (CLIENT span)
 └── subagent.invoke (INTERNAL)

The whole tree nests under one trace id, and span ids link each child to its parent.

Required attributes

Per the 2025-2026 semconv, an LLM span carries:

gen_ai.operation.name: "chat", "text_completion", "embeddings", "execute_tool", or "invoke_agent".
gen_ai.provider.name: "openai", "anthropic", "google", "azure_openai".
gen_ai.request.model: the requested model string (e.g. "gpt-4o-2024-08-06").
gen_ai.response.model: the model actually served.
gen_ai.usage.input_tokens and gen_ai.usage.output_tokens.
gen_ai.response.id: the provider's response id, for correlation.

Tool spans add:

gen_ai.tool.name: the tool identifier.
gen_ai.tool.call.id: the specific call id.
gen_ai.tool.description: the tool description (optional).

Agent spans add gen_ai.agent.name, gen_ai.agent.id, and gen_ai.agent.description.

Span kinds

SpanKind.CLIENT is for calls that cross a process boundary, like the LLM provider or an MCP server. SpanKind.INTERNAL is for the agent's own loop steps and tool execution.

Opt-in content capture

By default, spans carry metrics and timing, not prompts or completions. Large payloads and PII stay off. To include content you set OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental plus the specific content-capture env vars. Review carefully before enabling that in prod.

Events on spans

Token-level detail goes in as span events:

gen_ai.content.prompt for input messages.
gen_ai.content.completion for output messages.
gen_ai.content.tool_call for the tool call as recorded.

Events are time-ordered within a span, which gives you a detailed replay.

Exporters

OTel spans export to:

Jaeger / Tempo. OSS, on-prem.
Langfuse. Built for LLM observability, and visualizes token usage.
Arize Phoenix. Evals and tracing together.
Datadog. Commercial, and natively parses gen_ai.* attributes.
Honeycomb. Column-oriented and friendly to ad-hoc queries.

These backends speak OTLP, the wire format, so your instrumentation code does not need to know which one is on the other end.

Propagation across MCP

When an MCP client calls a server, inject the W3C traceparent header into the request. Streamable HTTP supports standard headers. Stdio has no HTTP headers, and the spec's 2026 roadmap discusses adding a _meta.traceparent field on JSON-RPC calls.

Until that ships, put the traceparent in the _meta of each request by hand and have the server log the trace id.

Metrics

Alongside spans, the GenAI semconv defines three histograms:

gen_ai.client.token.usage
gen_ai.client.operation.duration
gen_ai.tool.execution.duration

Use them for dashboards that don't need per-call detail.

AgentOps layer

AgentOps (founded 2024) specializes in GenAI observability. It wraps popular frameworks (LangGraph, Pydantic AI, CrewAI) to emit OTel spans automatically. That helps if your stack uses a supported framework. Otherwise, instrument by hand.

Check: The Concept0/28

1. What is OpenTelemetry (OTel)?Define

A card showing the abbreviation OTel expanded to OpenTelemetry, next to a small card listing where the name appears in this lesson: its title, its code and its reference links. — Name card only. The question asks what the thing is.

2. Which of these is listed as a required attribute for an LLM span?Define

A card for an llm.chat span listing operation name chat, provider name openai and request model gpt-4o, followed by a row saying more required attributes are not listed. — Three attributes copied from main.py; the rest of the span is left out.

3. What is a span in a trace?Define

A terminal-style box with one span summary: name llm.chat, an id, a duration of 50.27 milliseconds and two gen_ai attributes. — Reformatted excerpt of one real printed line from python main.py (one run); kind, trace, parent and other attributes left out.

4. What is a trace?Define

A table of three spans from a main.py run with columns name, trace and id: agent.invoke_agent, llm.chat and tool.execute, all with trace 1a6b51a1... and each with its own id. — Real values from one run, shortened as main.py shortens them.

5. What does a span's SpanKind tell a backend?Define

A terminal-style box showing the fields of one span record: name llm.chat, a hidden kind value, trace id, span id, parent span id, start and end times, attributes and events. — Field names from Span.to_otlp in main.py; ids shortened and the kind value hidden.

6. What is OTLP in an OTel setup?Define

A terminal-style box with the header line OTLP-JSON-shaped spans and one printed span line for agent.invoke_agent, reformatted over several lines, with its kind hidden. — Real output of one run, line broken for width; the kind value is hidden.

7. What is true about prompt and completion content capture by default?Define

Prompt string taken from main.py. Schematic, typed.

8. What does a traceparent carry?Define

Schematic, typed. Value elided.

9. What is an exporter in an OTel setup?Define

Schematic of the setup in the question, typed.

10. Which three attributes are listed for a tool-execution span?Define

A card for a tool.execute span listing operation name execute_tool, followed by a row saying the tool-span attributes are not listed. — One attribute copied from main.py; the rest left out.

11. What are gen_ai.content.prompt, gen_ai.content.completion, and gen_ai.content.tool_call?Define

Three rows naming gen_ai.content.prompt, gen_ai.content.completion and gen_ai.content.tool_call, each followed by a dashed box reading kind of item with a question mark. — Names copied from the lesson. Typed.

12. What does AgentOps do, per the AgentOps layer section?Define

A box for AgentOps, founded 2024, and a separate column of three framework boxes named LangGraph, Pydantic AI and CrewAI. No lines join them. — Names from the lesson. Typed.

13. Which kind of telemetry object is gen_ai.tool.execution.duration?Define

A card showing the name gen_ai.tool.execution.duration. — Name copied from the lesson. Typed.

14. Why does the llm.chat span use SpanKind CLIENT, while the agent's own loop steps use INTERNAL?Explain

Two cards: an llm.chat span with SpanKind CLIENT and operation chat, and the agent's loop steps with SpanKind INTERNAL. — Kinds as given in the stem. Typed.

15. Why do all spans in one agent run share a trace id and link to a parent by span id?Explain

A table of four spans with columns name, trace, id and parent in shuffled row order: llm.chat, tool.execute, agent.invoke_agent and llm.chat, all sharing one trace value. — Real values from one main.py run; rows shuffled to mimic a backend receiving spans unordered.

16. Why does an LLM span carry both gen_ai.request.model and gen_ai.response.model?Explain

A card for an llm.chat span listing operation name chat, provider name openai, and two attributes, gen_ai.request.model and gen_ai.response.model, whose values are hidden. — Attribute names from main.py; the two model values are hidden.

17. Why are prompts and completions left off spans unless you opt in?Explain

Three boxes in a row: an LLM call that has a prompt and a completion, a span named llm.chat, and a trace store that keeps spans. Arrows run left to right. — Schematic of the setup in the stem, typed.

18. Why must an MCP client inject a traceparent into the request it sends to a server?Explain

Schematic, typed. The trace id is shortened from one main.py run.

19. How should a trace cross an MCP stdio transport, given that stdio has no HTTP headers?Explain

A terminal-style box with a stdio JSON-RPC tools/call request for get_weather with city Tokyo and no headers, and a note saying the client writes it to the child process. — Typed example, not captured.

20. What do Jaeger, Langfuse, Arize Phoenix, Datadog, and Honeycomb have in common?Explain

A table of five backends with the lesson's short description of each: Jaeger or Tempo open source on-prem, Langfuse for LLM observability, Arize Phoenix for evals and tracing, Datadog commercial, Honeycomb column-oriented. — Descriptions condensed from the Exporters section.

21. When would you use the gen_ai.client.token.usage histogram instead of reading spans?Explain

A bar chart of simulated token counts across 200 calls, grouped into six ranges from 20-29 to 70-80 tokens, with the number of calls on each bar. — Simulated: 200 draws of random.randint(20, 80), the output-token range main.py uses. Not captured from a real metric.

22. A trace shows tool.execute lasting 28 seconds while its child mcp.call lasts 200 ms. Where should you look first?Analyze

A table of two spans: tool.execute under agent.invoke_agent lasting 28 seconds, and mcp.call under tool.execute lasting 200 milliseconds. — Values from the stem. Typed.

23. The agent's trace looks right, but each MCP server's spans appear as separate one-span traces with different trace ids. What is missing?Analyze

A trace list with four rows: one trace whose root is agent.invoke_agent and three separate traces with different ids, each holding a single server tools/call span. — The agent id is shortened from one run; the three server rows and their ids are illustrative.

24. After a staging agent is deployed, a privacy review finds customer names in the trace store. What is the likely cause?Analyze

Three boxes in a row: a deployed staging agent, a trace store holding its spans, and a privacy review that found customer names. — Schematic of the stem, typed.

25. Team A's MCP servers use Streamable HTTP and Team B's use stdio. How does trace propagation differ?Analyze

Two cards side by side. Team A, Streamable HTTP, and Team B, stdio, each show the transport name, a client to MCP server line and the same JSON-RPC body. — Typed illustration, not captured.

26. Support needs per-request latency and token cost from production, but the privacy team bans storing user text. Which setup fits?Design

Two cards: support needs per-request latency and per-request token cost from production, and the privacy team bans storing user text. — Requirements from the stem. Typed.

27. You add a sub-agent that the main agent can call. Where does it go in the trace?Design

A tree with agent.invoke_agent at the top and llm.chat, tool.execute and llm.chat beneath it, next to a dashed box reading sub-agent span: where? — Tree condensed from the lesson's hierarchy diagram; the sub-agent is not placed.

28. Product wants a dashboard of tool latency across all calls, and engineers want to inspect individual slow calls. What do you emit?Design

Two cards: product wants a dashboard of tool latency across all calls, and engineers want to inspect individual slow calls. — Needs from the stem. Typed.

Use It

code/main.py emits OTel-shaped spans to stdout (OTLP-JSON-like) for an agent that makes two LLM calls and three tool calls, each followed by one MCP round-trip. That adds up to nine spans in one trace. There is no real exporter here, and the printed lines are a summary of each span (name, kind, shortened ids, duration, gen_ai.* attributes, event count), not full OTLP. The point is the span shape and the attribute set, so read the output directly. The hierarchy diagram's subagent.invoke span is not emitted.

What to look at:

One trace id is shared across every span.
Parent-child links are encoded in parentSpanId. The mcp.call spans hang off tool.execute, and the LLM and tool spans hang off the root.
The required gen_ai.* attributes are populated on the LLM and tool spans.
Content capture is off by default. Run OTEL_CAPTURE_CONTENT=1 python code/main.py to turn it on and add the prompt and completion events. That variable belongs to the toy. A real setup uses the opt-in variables from the Opt-in content capture section.

Check: Use It0/6

1. main.py runs an agent with two llm.chat calls and three tool.execute calls, each tool followed by one mcp.call. How many spans print, counting the root?Apply

A code excerpt of agent_loop and fake_mcp_call from main.py, with arguments replaced by ellipses: a root span, a first llm.chat, a loop over three cities that starts tool.execute and calls fake_mcp_call, then a second llm.chat. — Lines condensed from code/main.py; span kinds and parents hidden.

2. In the main.py trace, what is the parentSpanId of each mcp.call span?Apply

A table of the first four spans of a main.py run with their span ids and parent ids: agent.invoke_agent, llm.chat, tool.execute, and mcp.call whose parent id is a question mark. — Real ids from one run, shortened.

3. How many spans in the main.py run have kind CLIENT?Apply

A code excerpt showing the Span class kind field with its comment, the start_span signature, and a list of the four span names started in the run. — Condensed from code/main.py (the start_span signature drops its type hints); the kind passed at each call is hidden.

4. You run main.py with OTEL_CAPTURE_CONTENT=1. What changes in the printed spans?Apply

A terminal-style box with the command OTEL_CAPTURE_CONTENT=1 python code/main.py and the first lines of its output, including content capture : ON. — Real output of the first lines; the rest is left out.

5. The mcp.call spans in main.py get a traceparent, yet it never appears in the printed output. Why?Apply

A terminal-style box with the printed summary line for an mcp.call span, reformatted: name, hidden kind, trace, id, hidden parent, duration, a shortened attrs object and zero events. — Real output of one run, broken for width; kind and parent hidden, attrs shortened.

6. What does the traceparent string built in fake_mcp_call contain?Apply

A terminal-style box showing one traceparent string of the form 00, a 32-character hex id, a 16-character hex id and 01. — Real value from one main.py run (a different run from the other pictures).

Ship It

This lesson produces outputs/skill-otel-genai-instrumentation.md. Give it an agent codebase and it returns an instrumentation plan: where to add spans, which attributes to populate, and which exporters to target.

Author's Note

I'd run main.py once with content capture off and once with OTEL_CAPTURE_CONTENT=1, and compare the two outputs. Nine spans print either way. With the variable set, the LLM spans also gain prompt and completion events, which shows how little code separates "metrics and timing" from "your users' text in the trace store."

The part worth reading closely is fake_mcp_call. It builds the traceparent from the MCP span's own trace id and span id and stores it as an attribute in the source, so you can see exactly what a server would need to receive. The printed output filters to gen_ai.* attributes, so the traceparent never shows up there. The toy never sends it either. It sits on the span as an attribute, not in a _meta field, so wiring it into a real request is the step you'd add yourself. That is also what Exercise 4 asks for.

I'd also keep the toy in proportion. Every span is hand-built, the LLM and the tools are fakes, and nothing computes the histograms from the Metrics section or sets gen_ai.tool.description or the agent description. In a real agent I'd reach for an OTel SDK and an OTLP exporter rather than maintain a span class, and keep this file as the reference for what the spans should contain.

Exercises

Run code/main.py. Count the spans and identify which is CLIENT vs INTERNAL.
Turn on content capture (env var) and confirm gen_ai.content.prompt and gen_ai.content.completion events appear. Note the implications for PII.
Add the tool-execution metric gen_ai.tool.execution.duration and emit it as a histogram sample per call.
Propagate a traceparent from a parent agent span into an MCP request's _meta.traceparent field. Verify the MCP server would see the same trace id.
Read the OTel GenAI semconv spec. Identify one attribute listed in the semconv that this lesson's code does NOT emit. Add it.

Key Terms

Term	What people say	What it actually means
OTel	"OpenTelemetry"	Open standard for traces, metrics, logs
GenAI semconv	"GenAI semantic conventions"	Stable attribute names for LLM / tool / agent spans
`gen_ai.*`	"The attribute namespace"	All GenAI attributes share this prefix
Span	"Timed operation"	A unit of work with a start, end, and attributes
Trace	"Cross-span ancestry"	Tree of spans sharing a trace id
SpanKind	"CLIENT / SERVER / INTERNAL"	Hints about span direction
OTLP	"OpenTelemetry Line Protocol"	Wire format for exporters
Opt-in content	"Prompt / completion capture"	Off by default; env var to enable
traceparent	"W3C header"	Propagates trace context across services
Exporter	"Backend-specific shipper"	Component that sends spans to Jaeger / Datadog / etc.