TAI BUI
Back to curriculum
LearnPython

Security — Secrets, API Key Rotation, Audit Logs, Guardrails

Keep secrets in a centralized vault (HashiCorp Vault, AWS Secrets Manager, Azure Key Vault) and out of config files, env files in VCS, and spreadsheets. Use IAM roles over static keys, and OIDC for CI/CD. The 2026 answer is the AI-gateway pattern: apps → gateway → model provider, with the gateway pulling credentials from the vault at runtime. Rotate in the vault and the apps pick up the new key within minutes, with no redeploys and no Slack threads asking who has the new key. Rotate on a ≤90 day policy and scan with TruffleHog / GitGuardian / Gitleaks on each commit. Zero-trust means MFA, SSO, RBAC/ABAC, short-lived tokens, and device posture. PII scrubbing uses entity recognition to mask PHI/PII before forwarding, and consistent tokenization (Mesh approach) maps sensitive values to stable placeholders so the LLM keeps code and relationship semantics. For network egress, put LLM services in a dedicated VPC/VNet subnet that allows only `api.openai.com`, `api.anthropic.com` and similar, and block other outbound traffic. The incident behind all this: a 2026 Vercel supply-chain attack used compromised CI/CD credentials to exfiltrate env vars across thousands of customer deployments.

TIME: ~60 minutesPREREQUISITES: Phase 17 · 19 (AI Gateways), Phase 17 · 13 (Observability)

Learning Objectives

  • Enumerate the four secret-management anti-patterns (config files in VCS, hardcoded env, spreadsheets, static keys) and name their replacements.
  • Explain the AI-gateway-pulls-from-vault pattern as 2026 production standard.
  • Implement a PII scrubber with consistent tokenization (same value → same placeholder) so semantics survive.
  • Name the 2026 Vercel supply-chain incident and what it taught about CI/CD credential hygiene.

The Problem

An intern commits .env with API keys and deletes it quickly. The keys are already in git history. A GitGuardian scan catches it, and your rotation process is "Slack the team, update 40 config files, redeploy all services." 8 hours later, half your services are live and half are waiting for deploy windows.

Separately, a user prompt includes "My SSN is 123-45-6789." The prompt goes to OpenAI. You have a BAA, but your internal policy says to mask PII before forwarding, and nothing did.

Separately again, your EKS cluster's LLM pod can reach any internet host. Someone exfiltrates data through DNS lookups to a domain they control, and nothing blocks it.

Security for LLM services has to cover all three vectors: vault-backed credentials, PII scrubbing, and network egress filtering. Audit logs tie them together by recording what happened.

Warm-up0/3
1. In an AI gateway setup, where does the gateway sit?Warm-up
A card naming AI gateway, tagged to the prerequisite AI Gateways lesson, with where it sits in the request path left as a question mark.
The term as named in the question. Where it sits is the question.
2. Which per-call data does LLM observability typically capture?Warm-up
A card naming LLM observability, tagged to the prerequisite Observability lesson, with what it captures per call left as a question mark.
The term as named in the question. What it captures is the question.
3. What does multi-factor authentication add to a password login?Warm-up
A password box marked entered correctly, an arrow, and a dashed box marked with a question mark for one more login step. A line says the site asks for one more step before it lets the user in and asks what that step is.
Setup: the password step, given. The extra step is left open.
Check: The Problem0/2
1. A teammate commits a `.env` holding a live provider key, then deletes it in the next commit. What is the correct response?Analyze
A two-commit git history: a commit that adds a .env file, then a commit that deletes it, with a note that the key was live and provider issued, and a line asking what the correct response is.
The two commits, given. The correct response is left open.
2. A key leaks and rotation takes 8 hours, with half the services still on the old key. Which missing ingredient is the root cause?Analyze
A sequence of three steps: key leaks, update 40 config files, redeploy all services, with a note that 8 hours later half the services are live and half still wait for a deploy window, and a line asking the root cause.
The rotation process, given. The root cause is left open.

The Concept

Centralized vault + IAM-role pull

Vault: HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, GCP Secret Manager. One source of truth.

IAM role: the app or gateway authenticates with its IAM identity instead of a static key. The vault returns the secret for the lifetime of the token.

The AI-gateway pattern: the gateway pulls OPENAI_API_KEY from the vault at request time. Rotate in the vault and the next request gets the new key. No redeploys.

Rotation policy ≤ 90 days

This covers API keys, vault root tokens, and CI/CD credentials. Automate rotation where you can. When rotation is manual, log it and track it.

Secret scanning

  • TruffleHog: regex plus entropy checks on commits.
  • GitGuardian: commercial, high accuracy.
  • Gitleaks: OSS, runs in CI.

Run them on each commit, and block the PR when a new secret is found.

Zero-trust posture

  • MFA on all accounts.
  • SSO via SAML/OIDC.
  • RBAC (role-based) or ABAC (attribute-based) for fine-grained access.
  • Short-lived tokens (hours, not days).
  • Device posture: corp devices with disk encryption only.

PII / PHI scrubbing

Before the prompt leaves your infra:

  1. Entity recognition (spaCy NER, Presidio, commercial).
  2. Mask matched entities: "My SSN is 123-45-6789" → "My SSN is [SSN_TOKEN_A3F]".
  3. Consistent tokenization (Mesh approach): the same value maps to the same placeholder, so the LLM can still follow relationships between entities.
  4. Optionally, a reverse mapping for the LLM response.

Static regex filters catch the basic patterns and NER catches more. Use both.

Input + output guardrails

Input: block known jailbreaks and forbidden topics, and rate-limit per user.

Output: regex-scrub for leaked secrets (API key patterns, email patterns in refusal contexts) and run a classifier for policy violations.

Network egress whitelist

Put LLM services in a dedicated subnet:

  • Whitelist: api.openai.com, api.anthropic.com, vector DB endpoints, vault endpoints.
  • Drop the rest.
  • Resolve DNS through an allowlist-only resolver to avoid DNS-tunneling exfil.

Audit log

An immutable log entry for each LLM call, with:

  • Timestamp.
  • User / tenant.
  • Prompt hash (not the raw prompt, for privacy).
  • Model + version.
  • Token counts.
  • Cost.
  • Response hash.
  • Any guardrail trips.

Retain per regulatory requirement (SOC 2 1 year, HIPAA 6 years).

The 2026 Vercel incident

A supply-chain attack: compromised CI/CD credentials exfiltrated env vars across thousands of customer deployments. CI/CD credentials are prod-equivalent, so store them in the vault, scope them narrowly, and rotate them aggressively.

Numbers you should remember

  • Rotation policy: ≤ 90 days.
  • Scan on each commit: TruffleHog / GitGuardian / Gitleaks.
  • Vercel 2026: CI/CD creds compromised, thousands of customer env vars leaked.
  • Audit log retention: SOC 2 = 1 year, HIPAA = 6 years.
Check: The Concept0/25
1. What is a vault in secrets management?Define
A card naming Vault, tagged to The Concept section's Centralized vault part, with what it is left as a question mark.
The term as the lesson names it. What it is is the question.
2. What is an IAM role in this setup?Define
A card naming IAM role, tagged to The Concept section's IAM-role pull part, with what it is left as a question mark.
The term as the lesson names it. What it is is the question.
3. What does OIDC for CI/CD replace in a build pipeline?Define
A card naming OIDC for CI/CD, tagged to the Key Terms section, with what it replaces left as a question mark.
The term as the lesson names it. What it replaces is the question.
4. Which tools handle commit-time secret scanning?Define
A card describing a scan that runs automatically as code is pushed and looks for leaked keys and tokens before merge, with no tool named, and which tools do this left as a question mark.
The task the tools perform, with no tool named. Which tools is the question.
5. How does RBAC differ from ABAC?Define
A card naming RBAC and ABAC side by side, both tagged to the Zero-trust posture part of The Concept, with how they differ left as a question mark.
Both terms as the lesson names them. How they differ is the question.
6. What does PII scrubbing do to a prompt before it leaves your infrastructure?Define
A card naming PII scrubbing, tagged to The Concept section's PII/PHI scrubbing part, with what it does to a prompt left as a question mark.
The term as the lesson names it. What it does is the question.
7. Two prompts mention the same email address. What does consistent tokenization do with it?Define
A card stating that prompt 1 mentions one email address and prompt 2 mentions the same email address, with what each prompt's placeholder is left as a question mark.
The given setup, exactly as stated in the question.
8. What is the Mesh approach in PII scrubbing?Define
A card naming Mesh approach, tagged to The Concept section's PII/PHI scrubbing part, with what it is left as a question mark.
The term as the lesson names it. What it is is the question.
9. What is an egress whitelist?Define
A card naming Egress whitelist, tagged to The Concept section's Network egress whitelist part, with what it is left as a question mark.
The term as the lesson names it. What it is is the question.
10. Which fields belong in the audit log entry for each LLM call?Define
A card naming Audit log, tagged to The Concept section's Audit log part, with what fields it stores left as a question mark.
The term as the lesson names it. What fields it stores is the question.
11. What rotation cadence do API keys, vault root tokens and CI/CD credentials need?Define
A card listing three things that need rotation: API keys, vault root tokens, and CI/CD credentials, with how often left as a question mark.
The things that need rotation, exactly as given. How often is the question.
12. Which set of controls makes up a zero-trust posture?Define
A card naming Zero-trust posture, tagged to The Concept section's Zero-trust posture part, with what controls make it up left as a question mark.
The term as the lesson names it. What makes it up is the question.
13. Which retention periods apply to audit logs under SOC 2 and HIPAA?Define
A card stating that SOC 2 has its own audit log retention number and HIPAA has its own, different number, with both numbers left as a question mark.
The two regulations named, exactly as given. The numbers are the question.
14. Why is an IAM role safer than a static API key baked into a deployment?Explain
Two boxes, a static API key baked into the deployment and an IAM role assumed by the app, with a line asking why the IAM role path is safer than the static key.
Setup: the two ways of holding a credential, given. Why one is safer is left open.
15. Why does the lesson say to use both static regex filters and NER for PII?Explain
Two boxes, regex filters and NER, both feeding into one PII scrub step, with a line asking why the lesson runs both instead of either alone.
Setup: that both feed the same step, given. Why both is left open.
16. What egress posture fits a subnet running LLM services?Explain
A card stating that a subnet runs LLM services and needs to reach the model provider's API, with what egress posture fits left as a question mark.
The subnet and its one stated need, given. The posture is the question.
17. Why does the audit log store a prompt hash instead of the raw prompt?Explain
A prompt flowing through a one-way hash into the audit log entry, with a line asking why the hash is stored and not the prompt itself.
Setup: that a hash, not the prompt, reaches the log, given. Why is left open.
18. What do input and output guardrails each do?Explain
Two cards, Input guardrails and Output guardrails, both tagged to The Concept section's Input and output guardrails part, each with what it does left as a question mark.
Both terms as the lesson names them. What each one does is the question.
19. What did the 2026 Vercel supply-chain incident teach about CI/CD credentials?Explain
A card stating that CI/CD credentials were compromised and env vars leaked across many customer deployments, with what it teaches about CI/CD credentials left as a question mark.
The incident's own given facts. The lesson drawn from it is the question.
20. A gateway pulls OPENAI_API_KEY from the vault on each request. The key is rotated in the vault at 10:00. Which key does a request at 10:01 use?Apply
A vault that pulls OPENAI_API_KEY on each request, with a timeline marking a key rotation at 10:00 and a request arriving at 10:01, with which key that request uses left as a question mark.
The rotation and the request, given on a timeline. Which key is used is the question.
21. An LLM pod in a subnet allowlisting only the provider, vector DB and vault hosts tries to reach attacker.example. What happens?Apply
An LLM pod inside a subnet that is allowed to reach only the provider, vector DB and vault boxes, with a separate box for attacker.example and a line asking what happens when the pod tries to reach that domain.
The allowlist and the attempted domain, given. What happens is left open.
22. An LLM pod's HTTPS egress is limited to approved hosts, yet data still leaks to an attacker's domain through DNS lookups. Which control is missing?Analyze
Two boxes, HTTPS egress limited to approved hosts, and an attacker's domain still reached through DNS lookups, with a line asking which control is missing.
The restricted HTTP path and the DNS leak, given. The missing control is left open.
23. Your CI provider was breached and you must stop jobs from holding long-lived cloud keys. Which design fits the lesson?Design
A card stating that the CI provider was breached and jobs currently hold long-lived cloud keys, with the design to fix this left as a question mark.
The breach and the current setup, given. The fix is the question.
24. You are building a gateway for a healthcare product that sends prompts to a model provider. Which design covers credentials, PII and compliance?Design
A card stating that a gateway for a healthcare product sends prompts to a model provider and must cover credentials, PII and compliance, with which design left as a question mark.
The product and its three stated concerns, given. The design is the question.
25. A vLLM deployment on EKS must call OpenAI, Anthropic and Weaviate. Which egress policy fits?Design
A card stating that a vLLM deployment on EKS must call OpenAI, Anthropic and Weaviate, with which egress policy fits left as a question mark.
The deployment and the three hosts it must call, given. The policy is the question.

Use It

code/main.py implements a toy PII scrubber with consistent tokenization, plus an audit-log entry builder that prints one JSON line per call.

Check: Use It0/4
1. A shared `Scrubber` from main.py has already scrubbed prompt 1. What does it return for "Please contact 123-45-6789 regarding account [email protected]."?Apply
The real _token_for rule from main.py: if the raw value is already a key in tokens, return the stored placeholder; otherwise assign a new one. A shared Scrubber has already scrubbed prompt 1, and prompt 2's text is given, with what scrub returns for it left as a question mark.
The real rule and the given prompt 2 text. What comes out is the question.
2. main.py scrubs three prompts with one `Scrubber`. Prompt 1 holds an SSN, email and phone, prompt 2 repeats the SSN and email, prompt 3 has a new SSN, email and phone. How many entries are in `scrubber.tokens`?Apply
A card stating prompt 1 holds a new SSN, email and phone, prompt 2 repeats only the SSN and email, and prompt 3 holds a new SSN, email and phone, with the number of entries in scrubber.tokens left as a question mark.
The three prompts' given contents. The entry count is the question.
3. A fresh `Scrubber` from main.py processes "Maria Lopez, SSN 123-45-6789, asked a question." What comes out?Apply
A card for a fresh Scrubber given the text Maria Lopez, SSN 123-45-6789, asked a question. with what scrub returns for it left as a question mark.
The given input, exactly as stated in the question.
4. After a restart of the scrubbing service, a different person's SSN comes out as [SSN_001], a placeholder used for someone else earlier. What is the cause?Analyze
A sequence of three steps, service running, restart, new SSN scrubbed, with a result noted as [SSN_001], the same placeholder used for someone else earlier, and a line asking the cause.
The sequence of events, given. The cause is left open.

Ship It

This lesson produces outputs/skill-llm-security-plan.md. Given your regulatory scope and current state, it plans the vault migration, scrubber, egress policy, and audit log.

Author's Note

I'd run main.py and read the token table before anything else. Prompts 1 and 2 share an SSN and an email, and both come out with the same [SSN_001] and [EMAIL_001]. That works because Scrubber.tokens is keyed on the raw value, so the second sighting hits the dict instead of bumping the counter.

Treat it as a sketch of the idea, not a scrubber. It is three regexes applied in order (SSN, then email, then phone) with no NER, so a name in the prompt passes straight through. The placeholders are counters like [SSN_001], not the hash-suffixed form shown in the body, and the table lives in memory on one Scrubber instance. A real gateway needs that mapping to survive restarts and stay consistent across replicas.

The audit side is thinner still. audit_log_call builds a JSON string and main prints it. Nothing appends to storage, so the append-only property from the docstring is something you would have to build. What it does get right is hashing the scrubbed prompt rather than the raw one, which keeps the SSN out of the log. Cost is a hardcoded 0.0012, and the token table print shows the first three characters of each value. I'd drop both before pointing this at real data. The phone regex also leaves a stray paren on "(202) 555-0150", which prints as "([PHONE_002]".

Exercises

  1. Run code/main.py. Send two prompts referencing the same SSN. Confirm both get the same placeholder.
  2. Design the network egress policy for a vLLM-on-EKS deployment calling OpenAI + Anthropic + Weaviate.
  3. You discover a key in git history (2 years old). What's the correct response: rotate the key, scrub history, or both? Justify.
  4. Your audit log grows 10 GB/day. Design retention tiers (hot 30d, warm 12mo, cold 6yr).
  5. Argue whether reverse-tokenization (substituting real values back into LLM response) is worth the complexity versus keeping placeholders visible.

Key Terms

TermWhat people sayWhat it actually means
Vault"secrets store"Centralized credential management service
IAM role"identity-based auth"Role assumed by app; returns short-lived creds
OIDC for CI/CD"cloud-issued tokens"No static keys in CI — identity via OIDC
TruffleHog / GitGuardian / Gitleaks"secret scanners"Commit-time secret detection
RBAC / ABAC"access control"Role-based vs attribute-based
PII scrubbing"data masking"Remove or tokenize sensitive entities
Consistent tokenization"stable placeholders"Same value → same token each time
Mesh approach"Mesh tokenization"Semantic-preserving tokenization pattern
Egress whitelist"outbound allowlist"Only permitted domains reachable
Audit log"immutable history"Append-only record for compliance

Further Reading

Code

Ships With