LearnPython

Prompt Caching and Semantic Caching Economics

**Pricing snapshot dated 2026-04.** Numeric claims below reflect vendor rate cards captured at this lesson's publication; verify against the linked docs before quoting them downstream.

TIME: ~60 minutesPREREQUISITES: Phase 17 · 04 (vLLM Serving Internals), Phase 17 · 06 (SGLang RadixAttention)

Learning Objectives

Distinguish L2 prompt/prefix caching (KV reuse at provider) from L1 semantic caching (LLM bypass on similar prompts).
Explain Anthropic's `cache_control` explicit marking and the two TTL options (5-min vs 1-hour) with their price multipliers.
Compute expected monthly savings given hit rate, prompt/response mix, and token prices.
Name the parallelization anti-pattern that inflates bills by 5-10x and the dynamic-content anti-pattern that collapses hit rate.

The Problem

You add prompt caching to your RAG service and the bill stays flat. You measure the hit rate and it is 7%. The prompts look static, but they are not. The system prompt includes the current date formatted to the minute, a request ID, and a randomized example reorder for diversity, so each request writes a new cache entry and reads nothing.

Separately, your agent runs ten parallel tool calls per user question. All ten reach the provider before the first cache write completes, which gives you ten writes and no reads. The bill lands at 5-10x what "with caching" was supposed to cost.

Caching is a protocol, not a flag. There are two layers, and each fails in its own way.

Warm-up0/3

1. In vLLM prefix caching or SGLang RadixAttention, what is reused when two requests share the same leading tokens?Warm-up

Two requests whose first four tokens match, shaded in a shared color, and whose remaining tokens differ. A line asks what is reused across the shared run. — Setup: two requests sharing a leading run of tokens. What gets reused is left open.

2. Why can a prefix cache only match on the start of a prompt, not on a shared chunk in the middle?Warm-up

Six positions, each with faint arrows reaching back to every earlier position. Request A and B diverge at position 4, then coincidentally match again later. A box asks whether that later match can be reused. — Setup: positions whose computation depends on everything earlier, plus a coincidental later match. Whether it is reusable is left open.

3. Two text embeddings have a cosine similarity close to 1. What does that tell you about the texts?Warm-up

Setup: a small angle between two vectors and its real cosine value. The angle is illustrative; the cosine is computed from it.

The Concept

L2: provider prompt/prefix caching

Provider stores the attention KV for a cacheable prefix and reuses it on the next request that matches the prefix. You pay a write cost once and reads are nearly free.

Anthropic (Claude 3.5 / 3.7 / 4 series): explicit cache_control marker in the request. You tag which blocks are cacheable. TTL: 5-minute (write costs 1.25x base) or 1-hour (write costs 2x base). Cache reads: $0.30/M on Claude 3.5 Sonnet vs $3.00/M fresh, which is 10x cheaper (docs.anthropic.com, as of 2026-04). Rates differ per model (Opus and Haiku are published separately), so check the live pricing page.

OpenAI: automatic caching for prompts ≥1024 tokens (platform.openai.com, 2026-04). No explicit flag. Cached input is roughly 10x cheaper than fresh on current gpt-4o/gpt-5 rate cards. Neither the docs nor the release notes publish an official hit-rate baseline. Community reports cluster around 30–60% with careful prompt design. Monitor usage.cached_tokens to measure your own.

Google (Gemini): context caching via explicit API; the 1M-token context means caching pays off even more.

Self-hosted (vLLM, SGLang): Phase 17 · 06 covers RadixAttention, the same pattern on your own compute.

L1: app-level semantic caching

Before calling the LLM at all, hash the prompt, embed it, and look for a similar cached request (cosine similarity above a threshold, typically 0.95+). On a hit, return the cached response. On a miss, call the LLM and cache the result.

Open-source: Redis Vector Similarity, GPTCache, Qdrant. Commercial: Portkey Cache, Helicone Cache.

Vendor accuracy claims measure how often the returned cached response was semantically appropriate, which is a different number from how often you hit. Production hit rates run like this:

Open-ended chat: 10-15%.
Structured FAQ / support: 40-70%.
Code questions: 20-30% (small variants kill hits).
Voice agents repeating prompts: 50-80% (voice normalization fixed set).

The parallelization anti-pattern

Your agent makes 10 tool calls in parallel, and all 10 share the same 4K-token system prompt. The first cache write completes around 300 ms after the provider sees the prompt. Requests 2-10 arrive in the same millisecond window, so each one sees a miss. You pay 10 write premiums and collect 0 read discounts.

Fix: go sequential-first. Make request 1 alone, then fire 2-10 once its cache has populated. That adds about 300 ms to the first tool call and saves 5-10x on the bill.

The dynamic content anti-pattern

Your system prompt looks like:

You are a helpful assistant. The current time is 14:32:17.
User ID: abc123. Today is Tuesday...

Each request carries different bytes near the top, so each one writes a new entry and none of them hit.

Fix: put everything static in the cacheable prefix and append dynamic content after the cache boundary:

[cacheable]
You are a helpful assistant. [rules, examples, instructions]
[/cacheable]
[dynamic, not cached]
Current time: 14:32:17. User: abc123.

ProjectDiscovery moved from 7% to 74% cache hit rate this way and published the breakdown.

Stack batch + cache for overnight workloads

Batch APIs (Phase 17 · 15) give a 50% discount at 24-hour turnaround. Cached input stacks another ~10x on top. Overnight classification, labeling, and report generation can drop to ~10% of synchronous-uncached cost this way.

Numbers to remember

Pricing points are captured 2026-04 from the linked vendor docs and drift every few months, so re-check before relying on them.

Anthropic cached read: $0.30/M on Claude 3.5 Sonnet, roughly 10x cheaper than fresh input (docs.anthropic.com).
Anthropic cache write premium: 1.25x (5-min TTL) or 2x (1-hour TTL).
OpenAI auto-cache: applies to prompts ≥1024 tokens; cached input priced at roughly 10% of fresh input on current rate cards (platform.openai.com).
Semantic cache hit rate (community-reported): ~10% open chat, up to ~70% structured FAQ. Not a vendor-documented baseline.
ProjectDiscovery: 7% → 74% hit rate by moving dynamic content out of the prefix (project blog, 2025-11).
Parallelization anti-pattern: typical reports of 5–10x bill inflation when N parallel requests all miss the first cache write.

Check: The Concept0/29

1. What is an L2 prompt cache?Define

Inside a box labeled Provider: request 1 with a cacheable prefix runs a slow full compute and something gets stored, keyed by the prefix. Request 2 with the same prefix looks that something up and runs fast. What is stored is marked with a question mark. — Setup: one provider-side round trip that writes, then a second that reads. What gets stored is left open.

2. What is an L1 semantic cache?Define

A new prompt is embedded and compared to stored prompts in the application, in front of the LLM. A similar-enough match returns the stored response; otherwise the LLM is called and the result is stored. A box asks what this app-level layer is called. — Setup: the app-level embed-and-compare flow in front of the LLM. The name of the layer is left open.

3. Which Anthropic mechanism marks blocks as cacheable for L2 prompt caching?Define

A request block with text and type fields, plus a blanked attribute next to an ephemeral TTL value. A box asks for the attribute's name. — Setup: the shape of a cacheable request block, with the marking attribute's name blanked out.

4. What write-cost multipliers does Anthropic charge for the 5-minute and 1-hour cache TTLs?Define

Two TTL cards, 5-minute and 1-hour, each with an unknown write-cost multiplier. — Setup: the two TTL options. Both multipliers are left as question marks.

5. What is GPTCache?Define

Two named open-source app-level similarity stores, Redis Vector Similarity and Qdrant, next to a highlighted card for GPTCache whose own layer is left as a question mark. — Setup: two of the three named L1 tools, with GPTCache's own role left open.

6. What does cache hit rate measure?Define

Ten request outcomes in a row, each marked cache or compute, with a box asking what the fraction landing in the cache column is called. — Setup: a run of ten outcomes, cache or compute. The name for the resulting fraction is left open.

7. How does the parallelization anti-pattern inflate the bill?Define

One cache write in progress with five calls sharing the same prefix all arriving at it before it finishes. A box asks what happens to the five calls. — Setup: five same-prefix calls racing one in-progress write. The outcome is left open.

8. What is the dynamic-content anti-pattern in cacheable prefixes?Define

A system prompt with a changing timestamp and request id on top, then 3K tokens of fixed rules below, the whole block marked as the cacheable prefix. A box asks what is wrong with this order. — Setup: changing bytes placed above the fixed rules, inside the cacheable prefix.

9. What is RadixAttention?Define

A shared-prefix branch feeding three sequence tails on a self-hosted server. A box asks what this shared-prefix structure is called in SGLang. — Setup: three sequences branching off one shared prefix on self-hosted compute. The name is left open.

10. What cached-read price is given for Claude 3.5 Sonnet, compared with fresh input?Define

Two price cards, cached read and fresh input, both for Claude 3.5 Sonnet, with unknown per-million-token prices. — Setup: the two price categories. Both figures are left as question marks.

11. At what prompt length does OpenAI's automatic caching apply, and which field do you monitor for your own hit rate?Define

A prompt-length gauge with an unknown eligibility cutoff, and a note that a field in the response's usage object tracks cached tokens, with its name blanked. — Setup: the eligibility cutoff and the usage field name are both left open.

12. What does the ~0.95 value refer to in L1 semantic caching?Define

A gauge filled to 0.95, with a box asking what kind of number this is. — Setup: the 0.95 value on its own, with no label for what category of number it is.

13. How do production L1 semantic-cache hit rates compare for voice agents and code questions?Define

Voice agents repeating prompts next to code questions with small variants, both with unknown hit-rate ranges. — Setup: the two workloads named. Both hit-rate ranges are left open.

14. What is the difference between L1 semantic caching and L2 prompt/prefix caching?Explain

A new prompt splitting into two paths: one where the app checks similarity first and can skip the LLM call, another sent straight to the provider, which matches the prefix itself. A box asks which path is L1 and which is L2. — Setup: the two paths a new prompt can take. Which one is L1 and which is L2 is left open.

15. How do Anthropic's two cache TTL options trade off?Explain

A short bar for the 5-minute TTL next to a much longer bar for the 1-hour TTL, both with an unknown write cost. — Setup: the two TTLs drawn to relative scale. Both write costs are left as question marks.

16. Why does sending request 1 alone before requests 2-10 turn their misses into reads?Explain

Request 1 fires alone at t=0 on a timeline, with a write-in-progress bar of unknown duration. Nine small markers for requests 2-10 sit further along the same timeline, held until that bar ends. — Setup: request 1 alone, then requests 2-10 waiting. How long the write takes is left open.

17. A semantic-cache vendor advertises 95% accuracy. What does that tell you about your cache hit rate?Explain

An accuracy claim of 95% next to an unknown hit rate figure. A box asks whether one tells you the other. — Setup: the vendor's accuracy claim and the reader's own unknown hit rate.

18. How can batch and cached input stack to cut cost on an overnight pipeline?Explain

Setup: the starting cost. Both discounts and the combined result are left open.

19. How does Google Gemini handle L2 caching, and why does it matter for long contexts?Explain

A 1M-token context window next to an unknown L2 caching-support card for Gemini. — Setup: the context window size. Whether and how caching works is left open.

20. Why do code questions reach only 20-30% semantic-cache hit rates?Explain

Two near-duplicate code questions and a similarity gauge sitting just under a 0.95 reference mark, with a note that this workload's hit rate is 20-30% and a question asking why. — Setup: a near-duplicate pair scoring just under the similarity cutoff (illustrative), with the lesson's 20-30% figure for this workload given.

21. You send 200M fresh-priced input tokens a month at $3.00/M. A 50% L2 hit rate reads at $0.30/M, and misses pay $3.00/M (ignore write premiums and output). How much do you save per month?Apply

200M fresh-priced tokens a month next to the same volume split 50/50 between cached reads and fresh misses. The monthly saving is left as a question mark. — Setup: the stem's own numbers for both scenarios. The saving is left open.

22. A prefix costs 1.0 per uncached use and a cached read costs 0.1. Using the lesson's write multipliers, after how many requests does a 1-hour TTL first cost less in total than no caching, if every request after the first reads?Apply

An unknown 1-hour write multiplier next to a given 0.1 read cost. A line asks after how many total requests caching first costs less than no caching. — Setup: the stem's own read cost and uncached cost. The write multiplier and the break-even count are left open.

23. Your RAG service shows a 7% cache hit rate. The system prompt opens with "The current time is 14:32:17. User ID: abc123." and then 3K tokens of fixed rules and examples. What do you change first?Analyze

A system prompt opening with a timestamp and user id, followed by 3K tokens of fixed rules, both marked as part of the cacheable prefix. A box asks what to change first. — Setup: the exact prompt layout from the stem. The fix is left open.

24. An agent fans out 12 parallel tool calls per question, all with the same 4K-token system prompt marked cacheable. Reads stay at zero and the bill is about 10x the plan. Which cause fits?Analyze

Twelve small numbered boxes for twelve parallel calls sharing one 4K prefix, with a note that reads stay at zero and the bill is about ten times the plan. A box asks which cause fits. — Setup: the stem's own call count, prefix size and outcome. The cause is left open.

25. A team sends an identical 800-token prompt to OpenAI each time and usage.cached_tokens is always 0. What is the most likely cause?Analyze

An 800-token prompt sent identically every time, with usage.cached_tokens always reading 0. A box asks for the most likely cause. — Setup: the stem's own prompt size and the always-zero field. The cause is left open.

26. You must label 500K documents overnight with no latency limit, and every request shares one long instruction prefix. Which design cuts cost the most?Design

500K documents to label overnight with no latency limit, every request sharing one long instruction prefix. A box asks which design cuts cost the most. — Setup: the stem's own scenario. The design choice is left open.

27. An agent sends 10 parallel sub-queries per question with one shared 4K prefix, and about 300 ms of extra delay on the first call is acceptable. Which design fits?Design

Ten sub-query boxes sharing one 4K prefix, with a note that 300 ms of extra delay on the first call is acceptable. A box asks which design fits. — Setup: the stem's own scenario and its stated delay tolerance. The design is left open.

28. A given prefix is reused about once every 20 minutes. Which Anthropic TTL can ever produce a read here, and why?Design

Two uses of the same prefix, about 20 minutes apart on a timeline. A line asks whether each TTL can still produce a read at the second use. — Setup: the stem's own reuse gap. Whether either TTL can read is left open.

29. A semantic cache at a 0.95 threshold hits 20%. At 0.85 it hits 50% but returns some wrong cached responses. Which choice fits?Design

A 0.95 threshold at 20% hit rate next to a 0.85 threshold at 50% hit rate with some wrong responses. A box asks which setting fits. — Setup: both thresholds and their stated outcomes, from the stem.

Use It

code/main.py simulates L1 and L2 caching on a mixed workload of about 1,300 requests (500 arrival events, 40% of them parallel waves of five) and prints cost, L1 hits, L2 reads, and L2 writes for five configs: no caching, L2 with the parallel penalty, L2 with the penalty fixed, and L2 with a 30% and a 70% semantic-cache hit rate. Compare the second and third rows to see what the parallelization penalty costs you.

Check: Use It0/9

1. In main.py, one 4000-token request runs with caching disabled (3.00 per M input, 15.00 per M output, 200 output tokens). What does it add to cost?Apply

The real BASE_INPUT and BASE_OUTPUT constants from main.py, with l2_enabled false, 4000 prompt tokens and 200 output tokens given, and the added cost left as a question mark. — Setup: the real pricing constants and this request's sizes. The total is left open.

2. In main.py with the 5-minute TTL, an 8000-token request misses the L2 cache. What does it add to cost?Apply

The real write-cost branch from main.py's simulate(), with the 5-minute TTL, an 8000-token prompt and a first-touch miss given. The write rate and the added cost are left as question marks. — Setup: the real branch and this request's TTL and size. Both the write rate and the total are left open.

3. In main.py with L2 enabled and parallel_penalty True, a wave of five requests with the same uncached prefix arrives. How many writes and reads does it add?Apply

Five requests sharing one uncached prefix, all arriving while it is still unwritten, with the resulting write and read counts left as question marks. — Setup: a five-request wave hitting one cold, shared prefix.

4. In main.py, a request is an L1 hit. What does it add to cost and to the L2 counters?Apply

The real L1-hit branch from main.py's simulate(), ending in a continue statement. A line asks what happens to cost and the L2 counters below it. — Setup: the real L1-hit branch, code copied from main.py. What continue skips is left open.

5. Using LRUCache from main.ts with capacity 2, you call set(a), set(b), has(a), then set(c). Which keys remain?Apply

Four LRUCache calls in order: set a, set b, has a, set c, on a capacity-2 cache from main.ts. The surviving keys are left as a question mark. — Setup: the exact sequence of calls, code copied from the question. The surviving keys are left open.

6. A SemanticCache in main.ts has threshold 0.95, and the best stored entry scores 0.93 against the query. What does lookup return?Apply

A gauge at 0.93 sitting just under a reference mark at 0.95, for main.ts's SemanticCache. What lookup returns is left as a question mark. — Setup: the best stored score against the 0.95 threshold, both given in the question. The return value is left open.

7. With the toy embed() in main.ts, a stored prompt is "how do I reset my password" and the query is "How do I reset my password?". What is the similarity?Apply

The real lowercase-and-strip line from main.ts's embed() function, with the stored prompt and the differently-capitalized, punctuated query shown. The similarity is left as a question mark. — Setup: the real normalization line and the two strings. The similarity is left open.

8. In main.py, the rows with and without parallel_penalty differ by only about $1 on a $6 bill, far from the 5-10x the lesson cites. Why?Analyze

A bar chart from a real run of main.py: parallel penalty on costs about $6.84, penalty off costs about $5.85. A line asks why the gap is this small. — Captured by running the lesson's own main.py, unmodified. Why the gap stays small is left open.

9. You change ttl from 5min to 1hr in main.py and expect fewer L2 writes. The write count is unchanged and only the cost per write rises. Why?Analyze

Two otherwise-identical configs differing only in ttl, each with an unknown l2_writes count. A line asks why the counts do not move. — Setup: the only variable changed is ttl. Both write counts are left open.

Ship It

This lesson produces outputs/skill-cache-auditor.md. Give it a prompt template and a traffic pattern, and it audits cacheability and recommends a restructure.

Author's Note

I'd run main.py before trusting any of the dollar figures above. It prices about 1,300 requests (500 arrival events, 40% of them waves of five) at the Sonnet-class rates in the constants at the top of the file, and the second and third rows differ only in the parallel-penalty flag, so the gap between them is the cost of the N-write trap on that workload. It comes to about $1 on a $6 bill, nowhere near 5-10x, because only the first touch of each of the 12 prefixes can miss.

Keep the simulator in proportion, though. The L1 layer is a coin flip at a fixed hit probability, so the 0.95 threshold is passed in but never used, and nothing is embedded. The L2 set also has no expiry, which means the 5-minute and 1-hour TTLs only change the write price here and never evict anything. Exercise 3 is where TTL choice gets real, and you have to bring your own arrival rate.

My bet is that the dynamic-content fix is the one to do first. It costs a prompt edit, no new infrastructure, and the 7% to 74% move in the concept section came from exactly that. I'd reach for a semantic cache only after the prefix is clean, and I'd measure usage.cached_tokens or the Anthropic equivalent before and after rather than trusting my read of the template.

Exercises

Run code/main.py. Toggle the parallelization flag. How much does the bill change?
Your system prompt has a date. Move it out. Show before/after hit rate math.
Calculate break-even for 1-hour TTL (2x write) vs 5-minute TTL (1.25x write) given your request arrival rate.
Semantic cache at 0.95 threshold hits 20%. At 0.85 it hits 50% but you see incorrect cached responses. Pick the right threshold and justify.
You batch 10 parallel sub-queries per user question. Rewrite for cache-friendliness without adding end-to-end latency.

Key Terms

Term	What people say	What it actually means
L2 prompt cache	"prefix cache"	Provider stores KV for repeated prefix
`cache_control`	"Anthropic cache marker"	Explicit attribute marking cacheable blocks
Cache write premium	"write tax"	Extra cost for first miss-to-cache (1.25x or 2x)
L1 semantic cache	"embedding cache"	App-level hash-and-embed before calling LLM
GPTCache	"LLM caching lib"	Popular OSS L1 cache library
Cache hit rate	"hits / total"	Fraction of requests served from cache
Parallelization anti-pattern	"the N-write trap"	N parallel requests miss cache N times
Dynamic content trap	"the time-in-prompt trap"	Dynamic bytes in prefix kill hit rate
RadixAttention	"intra-replica cache"	SGLang's prefix-cache implementation