LearnPython

Inference Platform Economics — Fireworks, Together, Baseten, Modal, Replicate, Anyscale

The 2026 inference market is no longer GPU time rental. It has split into custom silicon (Groq, Cerebras, SambaNova), GPU platforms (Baseten, Together, Fireworks, Modal), and API-first marketplaces (Replicate, DeepInfra). Fireworks raised GPU rental by $1/hr on May 1, 2026, and a $4B valuation on 10T+ tokens/day says the volume-driven model works. Baseten closed a $300M Series E at $5B in January 2026. The positioning is easy to remember: Fireworks optimizes latency, Together catalog breadth, Baseten enterprise polish, Modal Python-native DX, Replicate multimodal reach, Anyscale distributed Python. By the end you will have a matrix you can hand a founder.

TIME: ~60 minutesPREREQUISITES: Phase 17 · 01 (Managed LLM Platforms), Phase 17 · 04 (vLLM Serving Internals)

Learning Objectives

Name the three market segments (custom silicon, GPU platforms, API-first) and map each vendor to a segment.
Explain why custom-engine claims matter less at the platform layer than DX, attribution, and SLAs, given that vLLM and SGLang cover about 80% of production open-source inference.
Compute effective cost per request across at least three vendors and explain when per-minute (Baseten, Modal) beats per-token.
Identify which platform is the right default for a given workload (serverless bursty, steady high-throughput, fine-tuned variants, multimodal).

The Problem

You evaluated the managed hyperscaler platforms and decided you need a narrower, faster provider: Fireworks for latency, Together for breadth, Baseten for a fine-tuned custom model. Now you have six real choices and the pricing pages don't line up. Fireworks shows $/M tokens, Baseten shows $/minute, Modal shows $/second, Replicate shows $/prediction. You can't compare them head-to-head without modeling the workload.

The business model behind each page differs too. Fireworks runs its own engine (FireAttention) on shared GPUs, so the per-token rate reflects their utilization curve. Baseten gives you Truss plus dedicated GPUs, and the per-minute rate reflects that exclusivity. Modal is Python serverless with per-second billing and sub-second cold starts for small models. The output is the same (an LLM response) but the cost function is different for each.

This lesson models all six and tells you when each one wins.

Warm-up0/3

1. In vLLM serving, what does PagedAttention manage?Warm-up

A serving pipeline from incoming requests through a GPU serving engine to token output, with a component named PagedAttention highlighted inside the engine, and a question about what it manages. — Schematic; PagedAttention's role is the question.

2. On a managed LLM platform, which billing model charges you for idle capacity?Warm-up

Two taxi fare styles side by side: one where the meter runs the whole ride even in traffic, and one where the fare only counts distance traveled, with a question asking which style charges for idle time. — Everyday metaphor, not a specific vendor.

3. Why does continuous batching raise GPU throughput compared with static batching?Warm-up

Six seats at a table, one shown empty and dashed, with a new guest waiting and a question about seating them immediately versus waiting for every seat to free up. — Everyday metaphor for how a batch fills.

Check: The Problem0/1

1. Why can you not compare the Fireworks, Baseten, Modal, and Replicate pricing pages side by side?Explain

Four boxes named Fireworks, Baseten, Modal and Replicate, each labelled pricing page, with a question asking whether they are directly comparable. — Vendor names taken directly from the question; their pricing units are not shown.

The Concept

The three segments

Custom silicon. Groq (LPU), Cerebras (WSE), SambaNova (RDU). Typically 5-10x faster decode than a GPU-based cluster on the same model. The per-token price is higher (Groq was ~$0.99/M on Llama-70B in late 2025), but it is unbeatable for latency-sensitive use cases. Groq is the production pick for voice agents and real-time translation.

GPU platforms. Baseten, Together, Fireworks, Modal, Anyscale. They run on NVIDIA (H100, H200, B200 in 2026) or sometimes AMD, and sit between raw GPU rental (RunPod, Lambda) and a hyperscaler managed service (Bedrock).

API-first marketplaces. Replicate, DeepInfra, OpenRouter, Fal. Broad catalog, pay-per-prediction or pay-per-second, and the pitch is time-to-first-call.

Fireworks: latency-optimized GPU platform

FireAttention engine (custom); marketed as 4x lower latency than vLLM on equivalent configs.
Batch tier at ~50% of serverless rate for non-interactive workloads.
Fine-tuned models are served at the same rate as the base model, which separates Fireworks from providers that charge a premium for your LoRA.
On-demand GPU rental went up $1/hour effective May 1, 2026. Volume pricing is negotiable at scale.
Financial signal: $4B valuation, 10T+ tokens/day handled.

Together: breadth-optimized

200+ models including open-source releases within days of upstream publication.
50-70% cheaper than Replicate on equivalent LLM models. The "AI Native Cloud" positioning is volume and catalog.
Inference + fine-tuning + training in one API.

Baseten: enterprise-polish-optimized

Truss framework: model packaging with dependencies, secrets, and serving config in one manifest.
GPUs from T4 through B200. Per-minute billing with reasonable cold-start mitigation.
SOC 2 Type II and HIPAA-ready, which makes it a common fintech and healthcare pick.
$5B valuation, January 2026 Series E ($300M from CapitalG, IVP, NVIDIA).

Modal: Python-native-optimized

Infrastructure-as-code in pure Python. Decorate a function with @modal.function(gpu="A100") and deploy with one command.
Per-second billing. Cold starts run 2-4s with pre-warming and under 1s for small models.
$87M Series B at a $1.1B valuation (2025). Strongest developer experience score in independent surveys.

Replicate: multimodal breadth

Pay-per-prediction. The default platform for image, video, and audio models.
Integration ecosystem (Zapier, Vercel, CMS plugins).
Less competitive on LLM per-token rates, but it wins on multimodal variety.

Anyscale: Ray-native

Built on Ray; RayTurbo is Anyscale's proprietary inference engine (competes with vLLM).
Best for distributed Python workloads where the inference step is one node in a larger graph.
Managed Ray clusters; tight integration with Ray AIR and Ray Serve.

Per-token versus per-minute: when each wins

Per-token fits a latency-insensitive, bursty workload because you only pay for what you use. Per-minute fits high, predictable utilization, since you beat per-token once you saturate the GPU.

Rough rule: above ~30% sustained utilization of a dedicated GPU, per-minute (Baseten, Modal) starts to beat per-token (Fireworks, Together). Below that, per-token wins because you avoid paying for idle. The toy comparator in Use It adds a reserved-minute floor to the per-minute vendors, so its crossover lands higher than this rule.

Custom engine is the real moat

Every platform above vLLM and SGLang claims a custom engine: FireAttention, RayTurbo, Baseten's inference stack. Custom-engine claims are mostly marketing. vLLM and SGLang together represent about 80% of production open-source inference, so the differentiators at the platform layer are DX, attribution, and SLAs.

Numbers you should remember

Fireworks GPU rental: $1/hr raise effective May 1, 2026.
Fireworks claim: 4x lower latency than vLLM on equivalent configs.
Together: 50-70% cheaper than Replicate on LLMs.
Baseten valuation: $5B (Series E, Jan 2026, $300M round).
Modal valuation: $1.1B (Series B, 2025).
Per-minute beats per-token above ~30% sustained utilization.

Check: The Concept0/24

1. What is the main pitch of API-first marketplaces such as Replicate, DeepInfra, OpenRouter, and Fal?Define

Four boxes named Replicate, DeepInfra, OpenRouter and Fal grouped under the label API-first marketplaces, with a question about their main pitch to a developer. — Names taken directly from the question.

2. What is custom silicon in the inference platform market?Define

A term card showing custom silicon, tagged as one of three market segments from The Concept section, with a question asking what it defines. — The term as the lesson names it; its meaning is the question.

3. What is FireAttention?Define

A Fireworks feature list with FireAttention highlighted among batch tier, fine-tuned at base rate, and the May 2026 GPU rental increase, with a question mark asking what FireAttention is. — Feature names only; none is defined in the picture.

4. What is Truss?Define

A Baseten feature list with Truss highlighted among per-minute billing and the Series E valuation, with a question mark asking what Truss is. — Feature names only; none is defined in the picture.

5. What does per-token pricing charge for?Define

A term card showing Per-token, tagged as one of three billing models from the Key Terms table, with a question asking what it charges for. — The term as Key Terms names it; its meaning is the question.

6. What does per-minute pricing charge for?Define

A term card showing Per-minute, tagged as one of three billing models from the Key Terms table, with a question asking what it charges for. — The term as Key Terms names it; its meaning is the question.

7. What is per-prediction pricing, and where is it common?Define

A term card showing Per-prediction, tagged as one of three billing models from the Key Terms table, with two questions asking what it charges for and where it is common. — The term as Key Terms names it; its meaning is the question.

8. What is a batch tier?Define

A term card showing Batch tier, tagged as a non-interactive pricing tier from the Key Terms table, with a question asking what it is. — The term as Key Terms names it; its meaning is the question.

9. Which platform is Python-native serverless, with per-second billing and 2-4s cold starts after pre-warming?Define

Clues taken directly from the question.

10. What is the Fireworks "fine-tuned at base rate" pricing differentiator?Define

A term card showing fine-tuned at base rate, tagged as a Fireworks pricing differentiator, with a question asking what it means. — The term as the lesson names it; its meaning is the question.

11. What is RayTurbo?Define

An Anyscale feature list with RayTurbo highlighted among managed Ray clusters and Ray Serve integration, with a question mark asking what RayTurbo is. — Feature names only; none is defined in the picture.

12. Which mapping of vendors to market segments is correct?Explain

Vendor names taken directly from the options; slots are unlabeled and in no particular order.

13. Groq charges more per token than GPU platforms (about $0.99/M on Llama-70B in late 2025). Why is it still the production pick for voice agents?Explain

A card for Groq, labelled custom silicon, priced at about 0.99 dollars per million tokens on Llama-70B in late 2025, noted as chosen anyway for voice agents and real-time translation, with a question about why the premium is worth it. — Price and use cases taken directly from the question and lesson text.

14. How does shipping a model on Modal differ from shipping it on Baseten?Explain

Setup only; the shipping method is the question.

15. Why is Anyscale the natural fit when inference is one step in a larger distributed Python pipeline?Explain

A four-step pipeline, load data, preprocess, inference and postprocess, with inference highlighted, and a question about why Anyscale fits this kind of pipeline. — Schematic pipeline; the inference step is the only one highlighted.

16. Why does per-minute billing beat per-token billing only above some sustained utilization of a dedicated GPU?Explain

A utilization gauge from 0 to 100 percent with a dashed crossover line near 30 percent and a marker at 22 percent, asking what differs on each side of the line. — The ~30% figure is the lesson's own rule of thumb.

17. For LLM-only traffic, how does Together compare with Replicate, and where does Replicate still win?Explain

Two cards, Together with 200+ models and an all-in-one API, and Replicate with pay-per-prediction billing and plugin integrations, with questions about LLM price and multimodal variety. — Features taken from each platform's own section.

18. Why are custom-engine claims such as FireAttention and RayTurbo mostly marketing at the platform layer?Explain

A bar showing vLLM and SGLang together covering about 80% of production open-source inference, with the remainder marked other engines, and a question about what this means for platform differentiation. — The ~80% figure is stated directly in the lesson's Concept section.

19. Which platform fits a regulated healthcare customer that needs SOC 2 Type II, a HIPAA-ready posture, and dedicated GPUs?Explain

Clues taken directly from the question.

20. A nightly 10M-token job that does not need interactive latency moves from Fireworks serverless ($0.90/M) to its batch tier at about 50% of that rate. What does it cost per night?Apply

A Fireworks serverless rate of 0.90 dollars per million tokens next to a batch tier box at about 50% of that rate, with a 10 million token overnight job and a question for the cost per night. — Rates and job size taken directly from the question.

21. A team with light, uneven traffic chose a dedicated per-minute GPU because "dedicated is cheaper." Their bill is far above their token volume would suggest. What was the wrong assumption?Analyze

A utilization gauge with a marker far to the left near 8 percent, well below the ~30 percent crossover line, for a team whose bill is far above what their token volume suggests, asking what assumption was wrong. — The ~30% figure is the lesson's own rule of thumb; the team's load is illustrative and low.

22. One custom 70B model keeps its dedicated GPU busy at about 80% of capacity all day, with no latency target beyond normal chat. Which billing approach should be the default?Design

A utilization gauge with a marker at 80 percent, well above the ~30 percent crossover line, for a model with no latency target beyond normal chat, asking for the default billing approach. — The 80% figure and the ~30% rule of thumb are both taken directly from the question and lesson text.

23. You serve five LoRA fine-tunes of one base model with spiky, latency-sensitive traffic and want no premium for the variants. Which platform and billing is the best default?Design

A box for one base model with 5 LoRA fine-tunes, with traffic described as spiky and latency-sensitive and a requirement of no price premium for any variant, asking for the best platform and billing. — Setup taken directly from the question.

24. A product combines chat, image generation, and speech-to-text, and a small team wants to reach a first working call quickly. Which split fits?Design

Three boxes, chat, image generation and speech-to-text, for a small team that wants to reach a first working call quickly, asking how the three calls should be split across platforms. — Modalities taken directly from the question.

Use It

code/main.py compares the six vendors on a synthetic workload across pricing models and reports $/day and effective $/M tokens. It runs two scenarios (2M and 100M output tokens/day), a per-token vs per-minute sweep from 5% to 100% utilization, and a cold-start table.

Run it and read the sweep. Fireworks at $0.90/M against Baseten at $0.55/min, with Baseten's 1,440-minute reserved floor, flips to Baseten between 50% and 75% utilization, at about 68% ($792 against $1,166.40 a day at full load). That is well above the ~30% rule of thumb, because the toy bills Baseten for a full day regardless of load. The vendor numbers are rough approximations, so change them to match your own quotes.

Check: Use It0/9

1. A support bot serves about 2M tokens a day in spiky bursts. Which billing model fits, and why?Apply

A request-rate curve across one day with several tall, narrow spikes and long flat stretches near zero in between, for a bot serving 2 million tokens a day, asking which billing model fits and why. — Shape is illustrative; the 2M tokens/day figure is from the question.

2. In Scenario B (100M output tokens/day), what does cost_per_day return for Modal at $0.48/min, 800k tokens/min, and a 60-minute floor?Apply

A table of Modal's own parameters: 0.48 dollars per minute, 800,000 tokens per minute saturated, a 60 minute reserved floor, and a 100 million token per day workload, asking what cost_per_day returns. — Values taken directly from the question.

3. In Scenario A (2M tokens/day), what does effective_rate report for Baseten at $0.55/min with a 1,440-minute floor and 900k tokens/min?Apply

A table of Baseten's own parameters: 0.55 dollars per minute, 900,000 tokens per minute saturated, a 1,440 minute reserved floor, and a 2 million token per day workload, asking what effective_rate returns. — Values taken directly from the question.

4. In the break-even sweep, Baseten has a 1,440-minute floor at $0.55/min and Fireworks is $0.90/M. What does the 50% utilization row show?Apply

Fireworks at 0.90 dollars per million tokens and Baseten at 0.55 dollars per minute with a 1,440 minute floor, with a utilization of 50% for this one row, asking for each vendor's daily cost. — Vendor rates taken directly from the question; only this one row is shown.

5. You set min_reserved_minutes_per_day to 0 for Baseten and rerun the sweep. What changes?Apply

Baseten's reserved-floor parameter shown changing from 1,440 minutes to 0, with a question about the effect on the sweep's winner at each utilization level. — The parameter change is taken directly from the question.

6. cold_start_penalty assumes 100 cold invocations a day. How much extra latency per day does it print for Baseten, with a 5.0 s cold start?Apply

Baseten's cold start of 5.0 seconds and 100 cold invocations a day, with a formula asking for the extra latency per day as cold start seconds times 100. — Values taken directly from the question.

7. A chat product moved its LLM traffic to Replicate for convenience. At 100M tokens and 500,000 predictions a day it pays about $3,000 a day, while Together would cost about $88. What explains the gap?Analyze

Two bars for the same traffic, Replicate at 3,000 dollars a day and Together at 88 dollars a day, asking what explains the gap. — Both figures taken directly from the question.

8. A colleague reads the script output and concludes that Together is the best platform because it is cheapest at 2M tokens a day ($1.76 against Fireworks at $1.80). What is wrong with that conclusion?Analyze

Two nearly equal bars, Fireworks at 1.80 dollars and Together at 1.76 dollars a day, next to the conclusion that Together is the best platform, asking what is wrong with it. — Both figures captured from a real run of main.py's Scenario A.

9. In the Use It sweep, Fireworks at $0.90/M is compared with Baseten at $0.55/min and a 1,440-minute floor. Where does Baseten win, and why is that above the ~30% rule?Analyze

Fireworks and Baseten's rates restated, with a question about where across utilization from 5% to 100% Baseten wins, and why that is above the lesson's ~30% rule. — Vendor rates taken directly from the question; no specific utilization row is shown.

Ship It

This lesson produces outputs/skill-inference-platform-picker.md. Given a workload profile, SLA, and budget, it picks the primary inference platform and names the runner-up.

Author's Note

I'd run main.py first and look at Scenario A before anything else. At 2M tokens/day the per-minute vendors with a reserved floor (Baseten, Anyscale) land at hundreds of dollars a day against under $2 on per-token, which makes the idle penalty obvious in one table.

The part I'd poke at is the reserved-minute floor on the Vendor dataclass. It is why the break-even sweep flips to Baseten between 50% and 75% utilization instead of near the ~30% rule above. Set min_reserved_minutes_per_day to 0 for Baseten, rerun, and the crossover disappears: Baseten wins at every utilization in the sweep. Whether a real contract has that floor depends on the vendor, so use it as a knob and check the quote.

I'd also keep the toy in proportion. Prices, tokens-per-minute, and cold-start times are hand-picked approximations, and the script has no latency model, so it can't tell you what Fireworks's speed is worth. Put your own numbers in VENDORS before trusting any row.

Exercises

Run code/main.py. At what sustained utilization does Baseten (per-minute) beat Fireworks (per-token) for a 70B model on one H100? Derive the crossover yourself and compare to the rule of thumb.
Your product serves image generation plus chat plus speech-to-text. Pick platforms for each modality and name the gateway pattern that unifies them.
Fireworks raises prices by $1/hr on your primary model. Model the blended cost impact if 40% of your traffic moves to batch tier (50% off).
A regulated customer requires SOC 2 Type II + HIPAA + dedicated GPUs. Which three platforms are viable and which one wins on FinOps?
Compare cost per 1,000 predictions for Llama 3.1 70B on Fireworks serverless, Together on-demand, Baseten dedicated, and Replicate API. Which is cheapest at 10 predictions/day? At 10,000?

Key Terms

Term	What people say	What it actually means
Custom silicon	"non-GPU chips"	Groq LPU, Cerebras WSE, SambaNova RDU, optimized for decode
FireAttention	"Fireworks engine"	Custom attention kernel; marketed at 4x lower latency than vLLM
Truss	"Baseten's format"	Model packaging manifest; dependencies + secrets + serving config
Per-token	"API pricing"	Charge by tokens consumed; pay for no idle
Per-minute	"dedicated pricing"	Charge by wall-clock GPU time; wins at high utilization
Per-prediction	"Replicate pricing"	Charge per model invocation; common for image/video
RayTurbo	"Anyscale engine"	Proprietary inference on Ray; competes with vLLM on Ray clusters
Batch tier	"50% off"	Non-interactive queue at reduced rate; common on Fireworks, OpenAI
Fine-tuned at base rate	"Fireworks LoRA"	Charge LoRA-served requests at base model's rate (differentiator)