CapstonePython (agent), TypeScript (Slack integration)

Capstone 06 — DevOps Troubleshooting Agent for Kubernetes

AWS's DevOps Agent went GA, Resolve AI published its K8s playbooks, NeuBird demoed semantic monitoring, and Metoro tied AI SRE to per-service SLOs. They all converged on one shape: an alert webhook fires, an agent reads telemetry, walks a graph of K8s objects, ranks root-cause hypotheses, and posts a Slack brief with approval buttons. Read-only by default, every remediation gated by a human. This capstone builds that agent, scores it on 20 synthetic incidents, and compares it against AWS's Agent on three shared cases.

TIME: 30 hoursPREREQUISITES: Phase 11 (LLM engineering), Phase 13 (tools and MCP), Phase 14 (agents), Phase 15 (autonomous), Phase 17 (infrastructure), Phase 18 (safety)

Learning Objectives

Design a read-only-by-default tool surface for an SRE agent, with destructive verbs on a separate MCP server behind a Slack approval token.
Model a Kubernetes cluster as a knowledge graph (ownership, scheduling, observation edges) and walk it outward from an alert to collect telemetry.
Rank root-cause hypotheses with an evidence score built from recency, specificity, citation count, and graph-path length.
Log considered versus executed commands, and score the agent on a 20-scenario synthetic incident suite for RCA accuracy and time-to-hypothesis.

Problem

An AI SRE agent is a service that takes an incident alert, investigates the cluster on its own, and hands a human a ranked root-cause hypothesis to approve or reject. Through 2025-2026 that became the standard pitch: agents triage, humans approve remediations. AWS DevOps Agent, Resolve AI, NeuBird, Metoro, and PagerDuty AIOps all ship this shape in production. The agent reads Prometheus metrics, Loki logs, Tempo traces, kube-state-metrics, and a knowledge graph of K8s objects, then produces a hypothesis with telemetry citations in under five minutes. It never runs a destructive command without explicit human approval through Slack.

Most of the hard work here is scoping and safety, not reasoning. The agent needs a read-only-by-default RBAC surface, a hardened MCP tool server, and an audit log of every command it considered next to every command it executed. It has to recognize when it is out of its depth and escalate. And it has to run cheaply enough that an OOM-kill cascade does not turn into a $5k agent bill.

Warm-up0/3

1. An autonomous agent will be pointed at a production cluster. Which default permission level is the safest starting point?Warm-up

2. Why put a human approval step in front of an agent's destructive actions?Warm-up

3. When debugging a production incident, which combination of telemetry do engineers usually consult?Warm-up

Check: Problem0/2

1. What does an AI SRE agent hand to a human after an alert?Define

2. Which pairing of source and signal is correct for this agent's telemetry?Define

Concept

A K8s knowledge graph is the data structure the agent reasons over. Nodes are K8s objects (Pods, Deployments, Services, Nodes, HPAs, PVCs) plus telemetry sources (Prometheus series, Loki streams, Tempo traces). Edges encode ownership (Pod -> ReplicaSet -> Deployment), scheduling (Pod -> Node), and observation (Pod -> Prometheus series). A kube-state-metrics sync keeps the graph fresh, and it is re-sampled on every alert.

When an alert fires, the agent starts at the affected object and root-causes outward. It walks edges, pulls the relevant telemetry slices (last 15 minutes), and drafts a hypothesis. Hypotheses are ranked by evidence: how many telemetry citations support one, how recent they are, how specific. The top 3 go to Slack with graph-path visualizations and approval buttons for remediation actions.

Remediation is gated. The default actions are all read-only. Destructive ones (scaling down, rolling back, deleting Pods) need Slack approval, and ArgoCD rollback hooks need an auth token the agent never holds. The audit log records every command the agent considered, not only the ones it executed, so a review can catch near-misses.

Check: Concept0/14

1. What is a K8s knowledge graph, as this agent uses it?Define

2. What does read-only-by-default mean for this agent's tool surface?Define

3. What does the audit log hold?Define

4. What is a Slack approval card?Define

5. What is a telemetry citation?Define

6. What is the kube-state-metrics sync for in this design?Define

7. Which mechanism stops the agent from rolling back a bad deploy on its own?Explain

8. Why does the knowledge graph include ownership edges such as Pod to ReplicaSet to Deployment?Explain

9. When an alert fires, how does the agent gather its evidence?Explain

10. Why does the audit log record commands the agent considered as well as those it executed?Explain

11. Why read the considered-versus-executed log before trusting a high RCA accuracy score?Explain

12. Your agent scores 90% RCA accuracy, but the audit log shows it considered argocd_rollback without approval in 6 of 20 scenarios. What does that tell you?Analyze

13. The Slack brief blames Node ip-10-2-3-4, but the affected Pods moved to another Node 20 minutes ago. Which part of the design is the likeliest cause?Analyze

14. On-call proposes letting the agent run the rollback itself when its top hypothesis scores very high, to cut MTTR. Which response fits this design?Design

Architecture

PagerDuty / Alertmanager webhook
           |
           v
     FastAPI receiver
           |
           v
   LangGraph root-cause agent
           |
           +---- read-only MCP tools ----+
           |                             |
           v                             v
   K8s knowledge graph              telemetry slices
     (Neo4j / kuzu)              Prometheus, Loki, Tempo
   ownership + scheduling          last 15m, scoped
           |
           v
   hypothesis ranking (evidence weight)
           |
           v
   Slack brief + approval buttons
           |
           v (approved)
   ArgoCD rollback hook / PagerDuty escalate
           |
           v
   audit log: considered vs executed, every command

Stack

Observability sources: Prometheus, Loki, Tempo, kube-state-metrics
Knowledge graph: Neo4j (managed) or kuzu (embedded) of K8s objects and telemetry edges
Agent: LangGraph with per-tool allow-list, read-only by default
Tool transport: FastMCP over StreamableHTTP; separate server for destructive tools behind approval gate
Models: Claude Sonnet 4.7 for root-cause reasoning, Gemini 2.5 Flash for log summarization
Remediation: ArgoCD rollback webhook, PagerDuty escalate, Slack approval card
Audit: append-only structured log (considered, executed, approved, outcome)
Deployment: K8s deployment with its own narrow RBAC role; separate namespace

Check: Stack0/1

1. An OOM-kill cascade floods Loki and the cost of each incident is climbing. Which design change fits this stack?Design

Build It

Graph ingestion. Sync kube-state-metrics into Neo4j/kuzu every 30s. Nodes: Pod, Deployment, Node, Service, PVC, HPA. Edges: OWNED_BY, SCHEDULED_ON, EXPOSES, MOUNTS, SCALES. Add telemetry overlay edges: OBSERVED_BY (a Pod is observed by a Prometheus series).
Alert receiver. A FastAPI endpoint that accepts PagerDuty or Alertmanager webhooks. Extract the affected object(s) and the SLO breach.
Read-only tool surface. Wrap kubectl, Prometheus query, Loki logql, and Tempo traceql through FastMCP. Every tool gets a narrow RBAC verb ("get", "list", "describe"). The default server has no "delete", "exec", or "scale".
Root-cause agent. LangGraph with three nodes. sample pulls the last-15-minutes telemetry slice, walk queries the graph for neighboring objects, and hypothesize drafts ranked root-cause candidates with telemetry citations.
Evidence scoring. Each hypothesis gets a weighted score: recency and specificity carry the most weight, then citation count (capped at 5), then the inverse of graph-path length (main.py uses 0.35, 0.35, 0.2, 0.1). Return the top 3.
Slack brief. Post an attachment with the hypothesis, the graph-path visualization (a subgraph image rendered server-side), and approval buttons for at most one remediation action.
Remediation gate. Destructive tools (scale down, roll back, delete) live on a second MCP server behind an approval token. The agent can call them only after a human approves the Slack card.
Audit log. Append-only JSONL. For every candidate command, record whether it was considered, whether it was executed, and who approved it. Ship it to S3 daily.
Synthetic incident suite. Build 20 scenarios: OOMKill cascade, DNS flap, HPA thrash, PVC fill, noisy neighbor, faulty sidecar, bad ConfigMap rollout, certificate rotation, image-pull backoff, and so on. Score the agent on root-cause accuracy and time-to-hypothesis.

Check: Build It0/21

1. How are root-cause hypotheses ranked before the Slack brief?Define

2. Why do destructive tools live on a second MCP server instead of the default one with a prompt telling the model not to use them?Explain

3. How does graph-path length affect a hypothesis's score?Explain

4. Which two factors carry the most weight in the evidence score, and what does that favor?Explain

5. Why build the read-only server, the gated destructive server and the audit log before the LangGraph agent?Explain

6. Why score the agent on a suite of synthetic incidents such as OOMKill cascade and DNS flap?Explain

7. In the sample cluster the alert is on Deployment/checkout-api. How many hypotheses does root_cause return, and what is missing?Apply

8. A hypothesis has recency 30 minutes, specificity 0.5, 7 citations and path length 1. Using the main.py weights, what is its score?Apply

9. After main() finishes the demo, how many audit entries exist and how many are marked executed?Apply

10. agent.call('kubectl_exec', {}, approver='alice@sre') runs on the demo Agent. What does it return?Apply

11. agent.call('promql', {}, approver='alice@sre') runs on the demo Agent. What does the audit entry show?Apply

12. A signed request reaches verifySlackSignature with a valid HMAC, but its timestamp is 400 seconds older than the server clock. What does it return?Apply

13. A /slack/command request has its body altered in transit but keeps the original signature header. What does the server answer?Apply

14. A /slack/interactivity request has a valid signature and a payload of {} with no actions. What reply text does the server return?Apply

15. In main.py's sample cluster, root_cause runs on Deployment/checkout-api. How many citations does the bad-rollout hypothesis carry?Apply

16. Two hypotheses match in recency, specificity and path length. One cites 5 telemetry sources and the other cites 9, yet their scores tie. What explains it?Analyze

17. The audit log shows kubectl_delete with executed true and approved false. What does that point to?Analyze

18. Design A gives the agent delete and scale tools and tells the model to ask for Slack approval first. Design B puts them on a second server behind an approval token. After a wrong hypothesis, how do they differ?Analyze

19. An attacker replays a captured, validly signed Slack approve request 20 minutes later. Which control rejects it?Analyze

20. You want the agent to restart a crashing Pod after approval. Where should the new tool go?Design

21. You add an expired-certificate scenario to the 20-scenario suite. What must it carry so the agent is scored the same way as the other scenarios?Design

Use It

webhook: alert.pagerduty.com -> checkout-api SLO breach, error rate 14%
[graph]   affected: Deployment checkout-api (3 Pods, Node ip-10-2-3-4)
[walk]    neighbors: ReplicaSet checkout-api-abc, Service checkout-api,
           recent rollout 14m ago
[sample]  prometheus error_rate 14%, up-trend; loki 500s on /api/v2/pay
[hypo]    #1 bad rollout: latest image checkout-api:v2.41 fails /healthz
          citations: deploy.yaml (rev 42), prometheus errorRate, loki 500 stack
[slack]   [ROLL BACK to v2.40]  [ESCALATE]  [IGNORE]
          (approval required; agent does not roll back unilaterally)

Check: Use It0/1

1. In the TypeScript skeleton, mockAgent receives the text "latency spike on checkout". What report comes back?Apply

Ship It

outputs/skill-devops-agent.md is the deliverable. Point it at a K8s cluster and an alert source and it produces ranked root-cause hypotheses plus a Slack-gated remediation flow.

Weight	Criterion	How it is measured
25	RCA accuracy on scenario suite	≥80% correct root cause across 20 synthetic incidents
20	Safety	Destructive-action guard never fires without Slack approval in the audit log
20	Time-to-hypothesis	p50 under 5 minutes from alert to Slack brief
20	Explainability	Every hypothesis has graph paths and telemetry citations
15	Integration completeness	PagerDuty, Slack, ArgoCD, Prometheus end-to-end working
100

Check: Ship It0/3

1. How do Key Terms define MTTR?Define

2. Why does the Ship It rubric require graph paths and telemetry citations on every hypothesis?Explain

3. In one incident the Slack brief appears 4 minutes after the alert, a human approves the rollback 10 minutes later, and the SLO recovers 2 minutes after that. What is the time-to-hypothesis for this incident?Apply

Author's Note

I'd build steps 3, 7, and 8 before the LangGraph agent. A read-only tool server, a second server for destructive tools behind an approval token, and the considered-vs-executed log are what make the agent safe to point at a real cluster. With them in place, a bad hypothesis cannot execute anything on its own, because a human approval sits between it and the cluster. My bet is that the considered-vs-executed log deserves a read before the accuracy number. A high RCA score tells you the hypothesis was right and says nothing about what the agent tried on the way there, and the near-miss audit in Exercise 2 is how you would find out.

Exercises

Run your agent on the same three incidents AWS's DevOps Agent is demoed on. Publish the side-by-side and report where your agent diverges.
Add a "near-miss" audit that flags any command the agent considered that would have been destructive without approval. Measure the near-miss rate over one week.
Swap the hypothesis model from Claude Sonnet 4.7 to a self-hosted Llama 3.3 70B. Measure the RCA accuracy delta and the dollar cost per incident.
Build a causal filter that separates correlated telemetry spikes from a true root cause. Train a small classifier on the 20-scenario labels.
Add a rollback dry-run: run an ArgoCD rollback against a staging cluster with the same manifest, so the rollback plan is verified on a live cluster before the Slack approval button appears.

Key Terms

Term	What people say	What it actually means
K8s knowledge graph	"Cluster graph"	Nodes = K8s objects + telemetry series; edges = ownership, scheduling, observation
Read-only-by-default	"Scoped RBAC"	Agent's service account has only get/list/describe verbs; destructive verbs live in a separate server behind approval
Audit log	"Considered vs executed"	Append-only record of every candidate command, whether it ran, who approved
Hypothesis ranking	"Evidence score"	Weighted sum of recency, specificity, citation count, and inverse graph-path length
Slack approval card	"HITL gate"	Interactive Slack message with remediation buttons; agent cannot proceed until a human clicks
Telemetry citation	"Evidence pointer"	A Prometheus query, Loki selector, or Tempo trace URL that supports a claim
MTTR	"Time to resolution"	Wall-clock from alert fire to SLO recovery