Capstone Lesson 28: Observability with OTel GenAI Spans and Prometheus Metrics

Observability, for an agent harness, means every model call, tool execution, and verification gate decision leaves a structured record you can query later, as traces and as metrics. A coding agent in production produces all three every turn, and none of them are useful without that telemetry. Three failure modes show up when it is missing or badly built.

The first is the missing trace. Something went wrong on Tuesday, and the only record is a 500-line chat log. Nothing says which tool ran, how long it took, how many tokens went into the prompt, or whether the gate refused anything. The agent author has to guess.

The second is the unparseable trace. The harness wrote spans but invented its own field names, so Grafana, Honeycomb, Jaeger, and the local CLI can't read them. Whatever tooling the team already runs goes to waste because the spans are non-standard.

The third is the unaggregated metric. One slow tool call is visible in the trace, but "what is the p95 latency of read_file calls over the last hour?" has no answer, because there are only traces and no metrics.

The OpenTelemetry GenAI semantic conventions exist for exactly this. They define a small set of standard attributes that span emitters across LLM frameworks share. If your harness writes those attributes, every OTel-compatible backend can read them.

A span is the record of one operation in the harness. It carries a trace id (the whole agent invocation), a span id (this one operation), a name such as gen_ai.chat or gen_ai.tool.execution, attributes that follow the GenAI conventions, a start and end time, and a status. Every operation in the harness produces one.

The GenAI conventions standardise the attribute keys: gen_ai.system (the provider, e.g. anthropic, openai), gen_ai.request.model (the model id), gen_ai.request.max_tokens, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.response.model, gen_ai.response.id, and gen_ai.operation.name. Tool spans add gen_ai.tool.name and gen_ai.tool.call.id.

The exporter writes JSONL, one JSON object per line. It is the simplest format downstream tooling can stream, grep, and import. A real OTel exporter would speak OTLP gRPC. The JSONL exporter here is the offline equivalent, and it exits zero on every workstation.

Metrics live next to traces. A counter increments on each tool call: tools_called_total{tool="read_file"}. A histogram records the observed latency: tool_latency_ms{tool="read_file"}. Both serialise into Prometheus text exposition format, the de-facto standard for pull-based metrics.

The span builder is a small class with a span(name, attributes=None, parent=None, kind="INTERNAL") method that returns a context manager. On enter it records the start time. On exit it records the end time, attaches an exception if one was raised, pushes the finalised span to the exporters, and, for spans carrying gen_ai.tool.name, updates the tool counter and latency histogram.

The metrics registry is two dicts. Each counter is a Counter dataclass per name whose values dict is keyed by a sorted label tuple and holds floats. Histograms keep raw samples in a list and turn them into Prometheus histogram buckets at exposition time.

main.py ships:

1. GenAISpan dataclass: trace_id, span_id, parent_span_id, name, attributes, start_unix_nano, end_unix_nano, status, status_message, events.
2. SpanBuilder class with a span(name, attributes=None, parent=None, kind="INTERNAL") context manager.
3. JSONLExporter class with export(span) that appends one line.
4. Counter and Histogram classes plus MetricsRegistry.
5. prometheus_exposition(registry), which produces text-format output.
6. Metric updates inside SpanBuilder.span: any span carrying gen_ai.tool.name increments tools_called_total and observes tool_latency_ms.
7. A demo that synthesises a complete agent invocation (a gen_ai.chat span around tool spans), writes traces.jsonl to a temp directory, prints the Prometheus exposition, and exits zero.

Ids come from uuid4: the trace id is 32 hex characters (16 bytes) and the span id is the first 16 hex characters (8 bytes), matching the sizes in OTel's W3C trace context. The exporter has no error handling of its own, so an IO error in export propagates to the caller.

The histogram uses a fixed bucket set, the OTel default for latency in milliseconds: 5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000, +Inf. Samples are stored as a list, and exposition computes per-bucket counts on demand.

The OTel Python SDK is a real dependency. It is also several thousand lines of code, with multiple processes for the OTLP exporter and a runtime cost that swamps a lesson budget. The hand-rolled version teaches the wire format. In production you wire the same attributes into the real SDK and get the OTLP exporter, batching, and resource detection for free.

The conventions are stable. The wire format emitted here will keep parsing in 2030 because OTel never breaks GenAI attribute names; they only add new ones.

Lesson 25 produced the gate chain, lesson 26 the sandbox, and lesson 27 the eval harness. Lesson 28 makes all three observable. Lesson 29 wraps every step of the end-to-end demo in spans and prints the Prometheus text at the end.

cd phases/19-capstone-projects/28-observability-otel-traces
python3 code/main.py
python3 -m pytest code/tests/ -v

The demo writes traces.jsonl into a directory made by tempfile.mkdtemp (it is not deleted afterwards), then prints all four spans (chat, read_file, run_tests, and a denied shell call), then prints the Prometheus exposition for the counters and histograms. The tests check four things: spans serialise round-trip, the canonical GenAI attributes are present, counters increment correctly, and the histogram exposition contains the expected bucket counts.