TAI BUI
Back to curriculum
LearnPython

Batch APIs — the 50% Discount as Industry Standard

Every major provider ships an async batch API with a 50% discount and a 24-hour turnaround window. OpenAI, Anthropic, Google, and most inference platforms (Fireworks batch tier, Together batch) follow the same pattern. Stack batch with prompt caching and an overnight pipeline whose shared prefix dominates the input can fall to roughly 10% of its synchronous-uncached cost; the three workloads in this lesson land between 17% and 41%. The rule is simple: if it isn't interactive, it belongs on batch. Content generation, document classification, data extraction, report generation, bulk labeling, catalog tagging: anything that tolerates 24-hour latency is money left on the table until it moves. The 2026 production pattern is to triage every new LLM workload into three lanes: interactive (synchronous with caching), semi-interactive (async queue with fallback), and batch (overnight, cached input stacked). Workloads that pretend to be interactive but tolerate minutes of latency waste most of their spend.

TIME: ~45 minutesPREREQUISITES: Phase 17 · 14 (Prompt & Semantic Caching)

Learning Objectives

  • Name the three provider batch APIs (OpenAI, Anthropic, Google) and the common 50% discount + 24h turnaround guarantees.
  • Compute the cost for stacking batch + cached-input on an overnight classification workload and compare to synchronous-uncached baseline.
  • Triage a workload into interactive / semi-interactive / batch and justify the lane.
  • Name the two traps: partial interactivity (user expects faster than 24h) and output-schema drift (batch file format differs per provider).

The Problem

Your team ships a nightly report generation pipeline. 50,000 documents: summarize each, cluster the summaries, draft an executive brief. Run synchronously, it takes 4 hours and costs about $1,050 a night (the price comes from this lesson's script, with a 4,000-token shared prefix, 2,000 tokens per document and 200 output tokens). Then you hear about batch APIs.

Batch gets you 50% off. You also turn on prompt caching for the system prompt, which is shared across all 50k calls. Stacked, the bill drops to about $255 a night, 24.3% of baseline, roughly a quarter. Same pipeline, three config changes.

Batch is the cheapest lever in the LLM cost toolkit that nobody pulls. The reason is mostly organizational. Teams think "real-time" when the actual SLA is "by morning." This lesson is about not leaving about three quarters of the bill on the table.

Warm-up0/3
1. A 4K-token system prompt is shared by thousands of requests. What does prompt caching change about the bill?Warm-up
One 4,000-token system prompt sent unchanged on thousands of separate requests. A line asks what turning on caching changes about the bill.
Setup: the same shared prefix sent on every call. What caching changes is left open.
2. What is the difference between a synchronous and an asynchronous API call?Warm-up
Caller A stands at a counter and waits for a reply before doing anything else. Caller B makes the same kind of request, then walks away and returns later for the result. A line asks the name for each style.
Setup: two callers, one blocked at the counter, one that leaves and comes back. The two style names are left open.
3. What does an SLA (service level agreement) specify?Warm-up
A provider's written promise to commit to a worst-case level of service for a kind of request, with the specific commitment left blank. A box asks what this kind of promise is called.
Setup: the shape of a provider's written commitment. The name for this kind of promise is left open.

The Concept

The three batch APIs

OpenAI Batch API. You upload a JSONL file with a list of requests. The promised turnaround is 24 hours, though in practice it is often 2-8 hours. You get 50% off input and output tokens, served through the /v1/batches endpoint. Cache-eligible inputs also get cached-input pricing on top.

Anthropic Message Batches. Also a JSONL upload, also 24 hours, also 50% off. It supports cache_control: cache writes are explicit, and reads happen automatically within the batch.

Google Vertex AI Batch Prediction. Input comes from BigQuery or GCS. The discount for Gemini is similar at 50%, and it plugs into Vertex pipelines.

Asynchronous, not slow

Batch means "I promise to return within 24 hours." It does not mean "this will take 24 hours." A typical P50 is 2-6 hours. The provider schedules your batch in off-peak windows when GPU inventory is underused.

Stack with caching

Take a 50k-document summarization with the same 4K-token system prompt:

  • Synchronous uncached: 50000 × ($input × 4000 + $output × 200) at full rates.
  • Synchronous cached: the system prompt is cached after the first write, and the remaining 49999 calls get 10x cheaper input.
  • Batch cached: everything above, plus a 50% discount on both read and write.

Batch plus cache can land near 10% of the synchronous uncached bill when the shared prefix dominates the input. Any overnight workload with a shared system prompt should at least price this out.

Workload triage

Interactive. The user waits for the response and TTFT matters. Use a synchronous call with prompt caching. You cannot batch this.

Semi-interactive. The user submits a task and checks back in minutes. Use an async queue with fallback to sync when batch isn't available. Moderate-volume RAG indexing is a good example.

Batch. The user expects results "by morning" or "next hour": content pipelines, classification at scale, offline analysis. Always batch, and always stack caching on top.

The common mistake is calling everything interactive because the pipeline is in production. Production is not a latency spec. The SLA is.

The partial-interactivity trap

Some features look interactive but tolerate 5-10 minutes. Take a nightly customer health report with a "refresh" button. The user clicks refresh and a 10-minute wait is fine, but the team ships it synchronously. 50 concurrent refreshes then cost about 10x what a batched report delivered by email would.

Ask this: "What does 24 hours mean for this user?" If the answer is "they wouldn't notice," batch it.

The output-schema trap

Batch file formats differ per provider:

  • OpenAI: JSONL, one request per line.
  • Anthropic: JSONL, one message per line, with the response format embedded.
  • Vertex: a BigQuery table or GCS prefix with TFRecord.

A single batch client across providers means adapter code for each one. Gateways that advertise multi-provider batch (Portkey, and some LiteLLM tiers) still thin-wrap the raw format.

Numbers you should remember

  • Batch discount across providers: 50% flat on input and output.
  • Turnaround SLA: 24 hours guaranteed, 2-6 hours typical P50.
  • Stacked batch + cached input: roughly 10% of sync uncached cost.
  • Triage rule: if 24h latency is acceptable, always batch.
Check: The Concept0/20
1. What is the common batch-API offer across OpenAI, Anthropic, and Google in 2026?Define
Three provider cards, OpenAI Batch API, Anthropic Message Batches and Vertex AI Batch Prediction, each with a blank discount and a blank turnaround field. A line asks which two fields match across all three and what the shared values are.
Setup: three provider cards with matching blank fields. The shared values are left open.
2. What does "24-hour turnaround" actually guarantee?Define
A 24-hour line with the promised deadline marked at the far end, and a dashed, unlabeled bracket somewhere earlier on the line marked with a question for when delivery actually happens most of the time.
Setup: the 24-hour promised deadline on a timeline. Where delivery actually tends to land is left open.
3. What is JSONL, as used for batch requests?Define
A single file with three lines, each a complete, standalone JSON object for one batch request. A line asks what a file shaped like this is called.
Setup: a file with one complete JSON request per line. The name for this file shape is left open.
4. What is Anthropic's Message Batches?Define
Anthropic Message Batches drawn alone as a three-step flow, submit, an unlabeled processing step, results, with no other provider's product shown for comparison. A box asks what this product actually looks like.
Setup: Anthropic's own batch product in isolation. Its upload format, discount and turnaround are all left open.
5. What is Vertex AI Batch Prediction?Define
Vertex AI Batch Prediction drawn as a flow with an unlabeled input source feeding into Vertex processing and out to results. A note says this is Gemini's batch product and where its input comes from is open.
Setup: Vertex's own batch flow in isolation. Its input source is left open.
6. In batch work, what does the term "output schema" mean?Define
Two unnamed providers' result files drawn side by side, each a file-shaped box with differently arranged internal lines representing a different layout. A line asks what this difference between them is called.
Setup: two providers' result files, visibly differently shaped inside. The name for that difference is left open.
7. What is a stacked discount?Define
Two named discounts, batch discount and cached-input discount, shown as separate cards for one workload. A box asks what you call using both of them on one bill.
Setup: the two discounts named separately. What applying both together is called is left open.
8. What is workload triage?Define
A new workload forking into three unnamed lanes, lane A, lane B, lane C. A line asks what this three-way routing decision itself is called.
Setup: a workload splitting into three unnamed lanes. The name for the overall decision is left open.
9. What defines the semi-interactive lane in the workload-triage framework?Define
A single user submitting a task, waiting a few minutes, then checking back for the result. A line asks what is actually happening behind the scenes during that wait.
Setup: one task, one short wait, one check back. What handles it in the background is left open.
10. How does stacking batch with cached input change the bill versus synchronous uncached when the shared prefix dominates the input?Explain
A cost bar almost entirely taken up by one shared system prompt segment, starting at 100 percent, with an arrow down to a box asking for the result once both batch and caching apply.
Setup: a bill dominated by its shared prefix. The combined result is left open.
11. A team labels its nightly content pipeline interactive because it runs in production. What is wrong with that reasoning?Explain
A team's stated reasoning that a nightly content pipeline is interactive because it runs in production, with a note that the pipeline's own SLA is results by morning. A box asks what is wrong with that reasoning.
Setup: the team's own reasoning and SLA, both given. What is wrong with the reasoning is left open.
12. How does Anthropic's Message Batches API handle prompt caching within a batch?Explain
Two unresolved cards for an Anthropic batch, writing the cache entry and reading it later, each asking whether that step is explicit or automatic. A line asks whether the cache_control marking controls both.
Setup: the write step and the read step for one Anthropic batch. Whether each is explicit or automatic is left open.
13. How does a provider typically handle a batch you submit?Explain
A batch file flowing into an unlabeled provider box and out to results, with a note that neither safety checks nor model size change. A line asks what does change.
Setup: a batch already accepted by the provider. What happens inside the provider's own systems is left open.
14. Which statement about the three providers' batch offers is accurate?Explain
A three-row table, one row per provider, with discount, turnaround and file format columns all left blank. A line asks which column is the same across providers and which one actually differs.
Setup: a fully blank comparison grid. Which column matches and which one differs is left open.
15. What is the simplest decision rule for triaging a workload to batch?Explain
A single yes or no decision, whether the user would notice a 24-hour delay, branching to two unresolved outcomes.
Setup: the one decision question. Both outcomes are left open.
16. A team's batch client works for OpenAI. Why can't it submit the same file to Vertex unchanged?Explain
An OpenAI batch client's own request file, sent unchanged toward Vertex AI Batch Prediction. A line asks whether Vertex accepts that file as is.
Setup: the exact file an OpenAI client produces, aimed at a different provider. Whether it is accepted is left open.
17. A nightly customer health report has a "refresh" button. What goes wrong when the team ships it synchronously?Analyze
One user tolerating a 10-minute wait next to 50 users clicking refresh at once, all running synchronously. A line asks the cost impact compared to a batched version, as a multiple.
Setup: the single-user tolerance and the 50-concurrent-refresh scenario, both given. The cost multiple is left open.
18. A nightly job moves from OpenAI batch to Anthropic batch with the same prompts, and the result parser now fails on every line. What is the most likely cause?Analyze
OpenAI batch results that a parser handles correctly, next to Anthropic batch results from the same migrated job, where the same parser fails on every line. Both files are JSONL on the outside. A line asks what differs on the inside.
Setup: the migration from OpenAI to Anthropic batch, both JSONL on the outside, one parser failing on the new provider. What differs inside is left open.
19. A product has a "Generate monthly insights" button. Users click it and leave, and the results are emailed by the next morning. Volume spikes at month end. Which design fits?Design
A product feature description: a user clicks generate monthly insights and leaves the page, with results emailed by the next morning and volume spiking at month end. A box asks which design fits.
Setup: the feature exactly as described. Which design fits is left open.
20. You must run the same nightly pipeline on both OpenAI and Vertex batch. Which design fits?Design
One pipeline feeding into both OpenAI Batch and Vertex Batch Prediction. A line asks how to keep one codebase working for both providers.
Setup: one pipeline, two destination providers. How the codebase handles both is left open.

Use It

code/main.py computes costs across sync, sync+cache, batch, and batch+cache for three workloads, starting with the 50k-document one. It reports each configuration in dollars and as a percent of the sync baseline.

Check: Use It0/8
1. In main.py's 50k-document case (4,000-token prefix, 2,000 tokens per document, 200 output tokens), SYNC costs $1050.00. What does BATCH alone, with no cache, cost?Apply
The real cost_batch function from main.py, which multiplies cost_sync by BATCH_DISCOUNT, with the real SYNC total for the 50k-document case given and both the BATCH_DISCOUNT value and the resulting BATCH total left blank.
Setup: the real function and the real SYNC figure for this case. The discount's value and the BATCH total are both left open.
2. For main.py's 200k-item classification case (1,500-token prefix, 300 tokens per item, 50 output tokens), what percent of the sync baseline does BATCH + CACHE reach?Apply
The real SYNC total for main.py's 200k-item classification case, with the BATCH plus CACHE total and its percent of that baseline both left blank.
Setup: the real baseline for this exact case. The BATCH plus CACHE total and its share of baseline are left open.
3. In main.ts, a caller invokes dispatcher.addJob(batchId, input) after closeBatch(batchId) has started. What happens?Apply
A sequence of four real BatchDispatcher calls from main.ts in order: open, one job while queued, closeBatch starting, then one more addJob call right after close starts. A line asks what that last call does.
Setup: the exact call sequence from main.ts. What the final addJob call does is left open.
4. In main.ts, what does fakeClassifier return for the text "document body number 0" (22 characters)?Apply
The real fakeClassifier function body from main.ts, with the exact input text "document body number 0" given and the resulting label and confidence left blank for the reader to trace.
Setup: the real classifier code and the exact input text. Its label and confidence are left open for the reader to compute.
5. In main.ts costSyncCache, what does the first document (index 0) pay for a 4,000-token prefix?Apply
The real costSyncCache function from main.ts, showing that only the first document's loop iteration skips the cached-read branch. The CACHE_WRITE_5MIN multiplier and the resulting prefix cost for document 0 are both left blank.
Setup: the real function and the 4,000-token prefix size. The write multiplier and document 0's prefix cost are both left open.
6. A team's 50k-document run costs 24% of the sync bill, not the roughly 10% in the headline figure. What best explains the gap?Analyze
A bar chart of the 50k-document case's three token quantities per document: a 4,000-token shared prefix, 2,000 tokens of per-document text, and 200 output tokens. A note says this run lands at 24% of baseline rather than the headline figure, and asks which of the three keeps paying full rate.
Setup: the real per-document token split for this case. Which parts keep paying full rate is left open.
7. In the 50k case, SYNC + CACHE is 48.6% of baseline and BATCH is 50.0%. BATCH + CACHE reaches 24.3%. What does that show?Analyze
A bar chart of the 50k-document case's three given percentages, SYNC plus CACHE at 48.6%, BATCH at 50.0%, and BATCH plus CACHE at 24.3%. A line asks what multiplying the first two numbers shows about how the two discounts relate.
Setup: the three percentages, all given in the stem. What their arithmetic relationship shows is left open.
8. You plan to batch a workload with 20,000 tokens per document and a 500-token shared prefix. What expectation should you set for the stacked savings?Design
A gauge showing the shared 500-token prefix as a small share, about 2.4%, of one document's total tokens (the 500-token prefix plus 20,000 tokens of document text). A line asks what BATCH plus CACHE should land close to for this workload.
Setup: the real, tiny share the prefix makes up of this workload's tokens. The expected stacked result is left open.

Ship It

This lesson produces outputs/skill-batch-triager.md. Given workload characteristics, it triages the workload into interactive, semi-interactive, or batch and estimates the savings.

Author's Note

I'd run main.py before trusting the 10% figure, because the script doesn't reproduce it. Its 50k-document case, with a 4,000-token prefix, 2,000 tokens per document, and 200 output tokens, comes out at $1050.00 sync and $255.01 batch + cache, which is 24.3% of baseline. The classification case lands at 17.1% and the heavy-per-document report case at 41.3%. Only the shared prefix gets cached, so the per-document input and the output tokens keep paying full rate, and the stack gets close to 10% only when that prefix dominates the bill.

The prices in the script are hardcoded constants: $3.00 per million input tokens, $15.00 output, $0.30 cached input, and a cache write at 1.25x input. They are not any one provider's current price list, and the batch discount is a flat 0.50 multiplier over the whole cached bill. Cache expiry isn't modeled either. The constant is named CACHE_WRITE_5MIN, but the sync+cache function charges one write and then reads at the cached rate for every later document, with no refresh.

So I'd treat the printed percentages as shape, not forecast. Before moving a pipeline, I'd swap in my own prices and token counts and check the result against one real billed run. Exercise 1 asks for a 100k-document case the script doesn't include, which means adding a run(...) call to main().

Exercises

  1. Run code/main.py. For a 100k-doc pipeline with 3K-token system prompt and 500-token output, compute the savings of full stack (batch + cache) vs sync baseline.
  2. Pick three features in a real product you know. Triage each into interactive/semi/batch.
  3. A user complains their report took 3 hours. Was that a batch mis-triage or a legitimate interactive? Write the decision criterion.
  4. Your batch API return SLA is 24h but P99 is 20 hours. How do you communicate this to the user: what is the downstream system behavior on the edge case?
  5. Compute break-even: at what shared-prefix length does batch + cache become cheaper than running overnight on your own reserved GPU?

Key Terms

TermWhat people sayWhat it actually means
Batch API"async discount"50% off with 24h turnaround
JSONL"batch format"One JSON request per line; OpenAI/Anthropic standard
Message Batches"Anthropic batch"Anthropic's batch API product name
Batch prediction"Vertex batch"Vertex AI's batch API product
Turnaround SLA"24h promise"Guarantee, not typical; typical is 2-6h
Workload triage"interactivity decision"Interactive / semi / batch routing decision
Output schema"response format"Per-provider JSONL layout; not portable
Stacked discount"batch + cache"17-41% of the uncached sync bill in this lesson's runs; near 10% only when the shared prefix dominates

Further Reading

Code

Ships With