BuildPython

OpenTelemetry GenAI Semantic Conventions

OpenTelemetry's GenAI SIG (launched April 2024) defines the standard schema for agent telemetry. Span names, attributes, and content-capture rules converge across vendors, so an agent trace means the same thing in Datadog, Grafana, Jaeger, and Honeycomb.

TIME: ~60 minutesPREREQUISITES: Phase 14 · 13 (LangGraph)

Learning Objectives

Name the GenAI span categories: model/client, agent, tool.
Distinguish `invoke_agent` CLIENT vs INTERNAL spans and when each applies.
List the top-level GenAI attributes: provider name, request model, data-source ID.
Explain the content-capture contract: opt-in, `OTEL_SEMCONV_STABILITY_OPT_IN`, external-reference recommendation.

The Problem

Every vendor invents its own span names. The result is per-framework dashboards, one for each way of spelling "an agent called a tool." OpenTelemetry's GenAI SIG fixes this by defining one standard that the whole ecosystem targets, so you instrument once and read the same trace in any backend that supports the conventions.

Warm-up0/4

1. Why do teams add tracing to an agent system?Warm-up

2. Two vendors name the same agent operation differently in their traces. What problem does that cause?Warm-up

3. Why is it risky to write full user prompts into logs that many engineers can read?Warm-up

4. In an agent loop, what happens when the model requests a tool call?Warm-up

Check: The Problem0/1

1. What is the OpenTelemetry GenAI SIG?Define

The Concept

The OpenTelemetry GenAI semantic conventions are a standard set of span names, span kinds, and attribute keys for describing LLM and agent activity. Any instrumentation that follows them produces traces that compatible backends can interpret consistently, though how much each backend does with the attributes varies (see Stability).

Span categories

The conventions split agent telemetry into three categories:

Model / client spans cover raw LLM calls. Provider SDKs (Anthropic, OpenAI, Bedrock) and framework model adapters emit them.
Agent spans come in two kinds: create_agent fires when the agent is constructed, invoke_agent fires when it runs.
Tool spans are emitted once per tool invocation and connect to the agent span by a parent-child relation.

Agent span naming

The span name is invoke_agent {gen_ai.agent.name} when the agent has a name, and plain invoke_agent when it doesn't.

The span kind depends on where the agent lives:

CLIENT for remote agent services (OpenAI Assistants API, Bedrock Agents). Your process is calling out to someone else's agent.
INTERNAL for in-process agent frameworks (LangChain, CrewAI, a local ReAct loop). The agent runs inside your process.

Key attributes

gen_ai.provider.name: anthropic, openai, aws.bedrock, google.vertex.
gen_ai.request.model: the model ID you asked for.
gen_ai.response.model: the model that actually answered, which can differ from the request because of routing.
gen_ai.agent.name: the agent identifier.
gen_ai.operation.name: chat, completion, invoke_agent, or tool_call.
gen_ai.data_source.id: for RAG, which corpus or store was consulted.

Technology-specific conventions exist on top of these for Anthropic, Azure AI Inference, AWS Bedrock, and OpenAI.

Content capture

The default rule is that instrumentations SHOULD NOT capture inputs and outputs. Capture is opt-in, through three attributes:

gen_ai.system_instructions
gen_ai.input.messages
gen_ai.output.messages

The recommended production pattern is to store content externally (S3, your log store) and put references on the span: pointer IDs, not prose. This is the Lesson 27 content-poisoning defense wired into observability.

Stability

Most conventions are still marked experimental as of March 2026. The variable below opts you into the latest experimental version, so attribute names stay consistent across library upgrades:

OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental

Datadog v1.37+ maps GenAI attributes natively into its LLM Observability schema. Other backends (Grafana, Honeycomb, Jaeger) support the raw attributes.

Where this pattern goes wrong

Capturing full prompts in spans. PII, secrets, and customer data end up in traces that ops can read. Store content externally.
Omitting gen_ai.provider.name. Multi-provider dashboards break when attribution is missing.
Spans without parent links. You get orphaned tool spans. Always propagate context.
Skipping the stability opt-in. Your attributes may get renamed when the backend upgrades.

Check: The Concept0/28

1. Which three span categories does the GenAI SIG define?Define

2. Which span name represents an agent run in the GenAI conventions?Define

3. What is a CLIENT span in the GenAI conventions?Define

4. What is an INTERNAL span used for in the agent conventions?Define

5. What does the gen_ai.provider.name attribute record?Define

6. Which attribute identifies the corpus or store consulted for a retrieval?Define

7. Which values can the gen_ai.operation.name attribute take, per the lesson?Define

8. What does the gen_ai.agent.name attribute identify?Define

9. What does content capture mean in GenAI tracing?Define

10. Which three attributes carry opt-in content capture, per the lesson?Define

11. Which setting opts you into the latest experimental GenAI conventions?Define

12. What is the difference between the create_agent and invoke_agent spans?Explain

13. How are tool spans related to the agent span that invoked them?Explain

14. What happens if tool spans are emitted without propagating parent context?Explain

15. When does an invoke_agent span use kind CLIENT vs INTERNAL?Explain

16. Why might gen_ai.request.model and gen_ai.response.model differ on the same span?Explain

17. What breaks if a span is missing gen_ai.provider.name?Explain

18. How do the technology-specific conventions for Anthropic, Azure AI Inference, AWS Bedrock and OpenAI relate to the general GenAI conventions?Explain

19. What does Datadog v1.37+ do with GenAI semantic convention attributes, per the lesson?Explain

20. What is the default rule for content capture (inputs and outputs)?Explain

21. What does the lesson recommend for storing prompt content in production?Explain

22. An in-process CrewAI agent named "planner" runs inside your service. How should its agent-run span be recorded?Apply

23. Your code calls a remote Bedrock agent that has no name configured. What span records that call?Apply

24. After a second model provider is added, the cost-by-provider dashboard cannot attribute the new service's calls to any provider. What was left out of its instrumentation?Analyze

25. A team built dashboards on experimental GenAI attributes. After a backend upgrade several panels went empty because the attribute keys had been renamed. Which step did the team skip?Analyze

26. A developer records the invoke_agent span of an in-process LangChain agent as CLIENT, reasoning that the agent calls an LLM. What is the wrong assumption?Analyze

27. A service calls a remote Bedrock agent and also runs its own local tool functions, and the prompts contain customer data. Which instrumentation follows the conventions?Design

28. A router sends calls to either Anthropic or Bedrock and may swap in a different model than the one requested. You want cost dashboards by provider and by the model that actually served each call. What do you group by?Design

Build It

code/main.py is a stdlib span emitter that follows the GenAI conventions. It contains:

Span, carrying the GenAI attribute schema.
Tracer, with start_span and nested contexts.
A scripted agent run that emits create_agent, invoke_agent (INTERNAL), per-tool spans, and chat spans for the LLM calls.
A content-capture mode that stores prompts externally and records only IDs on the spans.

Run it:

python3 code/main.py

The output is a span tree with every required GenAI attribute, plus an "external store" that shows the content references recorded under opt-in capture.

Check: Build It0/4

1. In main.py the scripted model returns a search_tool call only when its prompt contains "search". All three loop prompts read "turn N: next action please". Which spans end up under invoke_agent research_bot?Apply

2. In main.py with capture_inline=False, ids are issued as content_001, content_002 and so on, in the order content is stored. The first chat span stores its input, then its output. Which attributes does that span carry?Apply

3. If main.py built its tracer with capture_inline=True instead, what would the first chat span hold?Apply

4. In main.py, create_agent research_bot is started and ended before invoke_agent research_bot starts. Where does the create_agent span sit in the tree?Apply

Use It

Where the conventions pay off:

Datadog LLM Observability (v1.37+) maps the attributes natively.
Langfuse / Phoenix / Opik (Lesson 24) auto-instrument the ecosystem.
Jaeger / Honeycomb / Grafana Tempo take raw OTel traces, and you build dashboards from the GenAI attributes.
Self-hosted: run the OTel Collector with a GenAI processor.

Ship It

outputs/skill-otel-genai.md wires OTel GenAI spans into an existing agent, with content-capture defaults and external-reference storage already set.

Author's Note

Treat the content-capture default as a feature, not a gap to fill. Spans are readable by everyone with access to your tracing backend, so the safe move is to leave prompts off the span and store them somewhere with proper access control, with a pointer ID on the span. You keep the content for debugging and you keep customer data out of the trace UI.

The other habit worth building early is setting gen_ai.provider.name on every span from day one. It's one attribute, and the multi-provider dashboards you'll want later depend on it. Pair it with the stability opt-in so a backend upgrade doesn't quietly rename the attributes you built those dashboards on.

Exercises

Instrument your Lesson 01 ReAct loop with invoke_agent (INTERNAL) + per-tool spans. Send to a Jaeger instance.
Add content capture in "references only" mode: prompts to SQLite, span attributes carry only row IDs.
Read the spec for gen_ai.data_source.id. Wire it into your Lesson 09 Mem0 search.
Set OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental and verify your attributes don't get renamed by the collector.
Build a dashboard: "which tool errors correlate with which models" from GenAI attributes alone.

Key Terms

Term	What people say	What it actually means
GenAI SIG	"OpenTelemetry GenAI group"	OTel working group defining the schema
invoke_agent	"Agent span"	Name of the span representing an agent run
CLIENT span	"Remote call"	Span for a call to a remote agent service
INTERNAL span	"In-process"	Span for an in-process agent run
gen_ai.provider.name	"Provider"	anthropic / openai / aws.bedrock / google.vertex
gen_ai.data_source.id	"RAG source"	The corpus or store a retrieval hit came from
Content capture	"Prompt logging"	Opt-in capture of messages; store externally in prod
Stability opt-in	"Preview mode"	Env var to pin experimental conventions