LearnPython

Managed LLM Platforms — Bedrock, Vertex AI, Azure OpenAI

The three hyperscalers made three different bets. AWS Bedrock is a model marketplace: Claude, Llama, Titan, Stability, and Cohere behind one API. Azure OpenAI is an exclusive OpenAI partnership, with Provisioned Throughput Units (PTUs) for dedicated capacity. Vertex AI is Gemini-first, with the strongest long-context and multimodal story. In 2026 Artificial Analysis measures Azure OpenAI at ~50 ms median and Bedrock at ~75 ms on Llama 3.1 405B equivalents, and PTUs explain the gap because dedicated capacity beats shared on-demand. The decision rule is not "which is fastest" but "which model catalog and FinOps surface match my product." Pick with the tradeoffs written down, not vibes.

TIME: ~60 minutesPREREQUISITES: Phase 11 (LLM Engineering), Phase 13 (Tools & Protocols)

Learning Objectives

Name the three platform strategies (marketplace vs exclusive vs Gemini-first) and match each to a product use case.
Explain what Provisioned Throughput Units (PTUs) buy you in Azure OpenAI and why on-demand Bedrock typically reads ~25 ms slower at the 405B scale.
Diagram the FinOps attribution surface for each platform (Bedrock Application Inference Profiles vs Vertex project-per-team vs Azure scopes + PTU reservations).
Write down a "two-provider minimum" policy and explain why single-vendor lock-in is the expensive mistake in 2026.

The Problem

You picked Claude 3.7 Sonnet for your product and now you need to serve it. You can call the Anthropic API directly, call it through AWS Bedrock, or go through a gateway. The direct API is the simplest. Bedrock adds BAAs, VPC endpoints, IAM, and CloudWatch attribution. A gateway adds failover, unified billing, and rate limits across providers.

The deeper question is catalog. If you need Claude and Llama and Gemini in the same product, you can't buy them all from one place unless that place is Bedrock plus Vertex plus Azure OpenAI at once. The hyperscalers are not interchangeable, because each one made a different bet on who owns the model layer.

This lesson maps the three bets, the latency gap, the FinOps gap, and the lock-in risk.

Warm-up0/3

1. What does time to first token (TTFT) measure for a streaming LLM call?Warm-up

A timeline from a request sent, to a first token arriving, to a stream of further tokens ending at response complete, with the gap from request to first token braced and labelled TTFT equals a question mark. — Generic streaming call, no specific service.

2. A service sends steady, high token volume around the clock. Which capacity model usually fits it best on a cost basis?Warm-up

A bar chart of one service's request volume across all 24 hours of a day, every bar nearly as tall as the next, with a question about which capacity model fits this traffic best on a cost basis. — Illustrative bars; traffic stays high around the clock with little idle time.

3. Your app calls one LLM provider directly and it starts returning errors during an outage. What does a gateway in front of several providers add?Warm-up

Two setups. Top, an app calling provider A directly during an outage with no response. Bottom, the same app going through a gateway that also reaches provider B, with a question about what the gateway adds. — Generic setup, no specific provider names.

The Concept

Three strategies

AWS Bedrock is the marketplace. Claude (Anthropic), Llama (Meta), Titan (AWS first-party), Stability (image), Cohere (embeddings), Mistral, plus image and embedding sub-catalogs. One API, one IAM surface, one CloudWatch export. Bedrock's bet is that customers want optionality more than they want a single model.

Azure OpenAI is the exclusive partnership. You get GPT-4 / 4o / 5 / o-series, DALL·E, Whisper, and fine-tuning of OpenAI models in Azure datacenters. The "Azure OpenAI Service" catalog has no non-OpenAI models; those go to Azure AI Foundry, a separate product. Azure's bet is that OpenAI remains the frontier and customers want enterprise controls on that specific relationship.

Vertex AI puts Gemini first and everything else second. Gemini 1.5 / 2.0 / 2.5 Flash and Pro, plus Model Garden for third-party models. Vertex's bet is multimodal long context, and the 1M-token Gemini window is the differentiator.

Latency gap at scale

Artificial Analysis runs continuous benchmarks. On equivalent Llama 3.1 405B deployments (shared on-demand), Azure OpenAI median first-token latency is around 50 ms and Bedrock is around 75 ms. The gap is not an AWS failure. It comes from a difference in capacity model. Azure sells PTUs (Provisioned Throughput Units), which reserve GPU capacity for your tenant. Bedrock has an equivalent (Provisioned Throughput), but it starts around $21/hour per unit, and most customers stay on shared on-demand.

On-demand shared capacity competes with every other customer's traffic. Dedicated capacity does not. If your product SLA is TTFT < 100 ms at P99, you either buy PTUs on Azure, buy Bedrock Provisioned Throughput, or accept the default variance.

Provisioned Throughput economics

Azure PTUs are a reserved block of inference compute. You can save up to ~70% versus on-demand for predictable workloads. The cost is fixed per hour regardless of traffic, so you pay for the reservation even when idle. The break-even is usually around 40-60% sustained utilization.

Bedrock Provisioned Throughput runs $21-$50 per hour depending on model and region. The math is similar, with break-even around half of peak utilization, and it needs a monthly commitment.

Vertex provisioned capacity is sold per Gemini SKU. Pricing varies by model and region and is less publicly advertised.

FinOps surface: the real differentiator

Bedrock Application Inference Profiles are the cleanest attribution in the marketplace. Tag a profile with team, product, feature, route all model invocations through it, and CloudWatch breaks out cost per profile without post-processing. They were added in 2025 and are still the most granular native option among the hyperscalers.

Vertex attribution is project-per-team plus labels on everything. You model each team as a GCP project, put labels on every resource, and use BigQuery Billing Export + DataStudio for rollups. It's more work, but BigQuery gives you arbitrary SQL on the cost data.

Azure relies on subscription and resource-group scopes plus tags, with PTU reservations as a first-class cost object. Tags are inherited from resource groups, not requests, so per-request attribution requires Application Insights custom metrics or a gateway that stamps headers.

The pattern: Bedrock is cleanest native, Vertex is most flexible via BigQuery, and Azure is most opaque unless you instrument it.

Lock-in is the 2026 risk

Single-hyperscaler commitment was fine when one model dominated. In 2026 the frontier moves monthly: Claude 3.7 one quarter, Gemini 2.5 the next, GPT-5 the quarter after. Locking to one platform locks you out of two-thirds of the frontier.

The pattern working teams adopt is a two-provider minimum for any product-critical LLM call. Bedrock plus Azure OpenAI is the common pair, with Claude from one, GPT from the other, failover between them, and the same gateway in front. The cost uplift is small because the gateway routes to the cheaper path, and the availability uplift during outages (the Azure OpenAI January 2025 incident, the AWS us-east-1 outage) is decisive.

Data residency, BAAs, and regulated industries

Bedrock: BAAs in most regions, VPC endpoints, guardrails. A common fintech default. Azure OpenAI: HIPAA, SOC 2, ISO 27001, EU data residency. The enterprise-regulated default. Vertex: HIPAA, GDPR, data residency per region, and Google Cloud's compliance stack.

All three meet the basic checkbox. The differences are in data retention policies, how logs are handled, and whether abuse monitoring reads your traffic (opt-in by default on most, with opt-out available for enterprise).

Numbers you should remember

Azure OpenAI median TTFT on Llama 3.1 405B equivalents: ~50 ms (with PTUs).
Bedrock median TTFT on-demand: ~75 ms.
Bedrock Provisioned Throughput: $21-$50/hr per unit.
Azure PTU break-even: ~40-60% sustained utilization.
PTU savings vs on-demand at high utilization: up to 70%.

Check: The Concept0/23

1. What is AWS Bedrock?Define

AWS Bedrock drawn as a central box reading one API, one IAM surface, one CloudWatch export, with arrows down to six vendor-family boxes: Claude, Llama, Titan, Stability, Cohere and Mistral, and a question asking what AWS Bedrock is. — The vendor families Bedrock serves, from The Concept section.

2. What does the Azure OpenAI Service catalog contain?Define

A box labelled Azure OpenAI Service listing GPT-4, 4o, 5, o-series, DALL-E, Whisper and fine-tuning of these models, next to a separate dashed box labelled Azure AI Foundry, a separate product, with a question about what the service's catalog contains. — Catalog contents from The Concept section.

3. What is Vertex AI's platform strategy?Define

Two equally styled boxes, Gemini 1.5, 2.0, 2.5 Flash and Pro, and Model Garden for third-party models, with a question about Vertex AI's platform strategy. — The two catalog sections named in The Concept section, same styling for both.

4. What is a Provisioned Throughput Unit (PTU) in Azure OpenAI?Define

A card showing the term PTU, tagged with the lesson section Provisioned Throughput economics, asking what it actually means. — The term as the lesson names it; its meaning is the question.

5. Which Bedrock feature gives the cleanest native per-product cost attribution in CloudWatch?Define

A CloudWatch-style panel titled cost by product feature, with four unlabelled bars for feature A through D, a note that CloudWatch breaks this out with no manual post-processing, and a question about which Bedrock feature makes the view possible. — Scenario from the lesson's FinOps surface section; the feature name is the question.

6. What is Model Garden on Vertex AI?Define

A card showing the term Model Garden, tagged with the lesson section Three strategies, asking what it actually means. — The term as the lesson names it; its meaning is the question.

7. What does a two-provider minimum policy require?Define

A card showing the term Two-provider minimum, tagged with the lesson section Lock-in is the 2026 risk, asking what it actually requires. — The policy as the lesson names it; its requirement is the question.

8. What is a Business Associate Agreement (BAA) needed for in a managed LLM deployment?Define

A card showing the term BAA, tagged with the lesson section on data residency and regulated industries, with context about a managed LLM deployment handling patient data, asking what a BAA is needed for. — The term as the lesson names it; what it covers is the question.

9. What is abuse monitoring on a managed LLM platform?Define

A card showing the term Abuse monitoring, tagged with the lesson section on data residency and regulated industries, asking what it is. — The term as the lesson names it; what it is is the question.

10. Why can't you get Claude through the Azure OpenAI Service?Explain

The Azure OpenAI Service catalog box holding only GPT and DALL-E and Whisper, a dashed Claude box marked not here, with arrows showing Claude is reachable instead through AWS Bedrock and the direct Anthropic API. A question asks the reason Claude is missing from this one catalog. — Catalog membership facts from The Concept section.

11. Which strength is the main differentiator of Vertex AI?Explain

A card showing one Gemini context window holding 1,000,000 tokens of text, image, audio and video together. — Figure from The Concept section, Three strategies.

12. On Llama 3.1 405B, on-demand Bedrock reads about 25 ms slower at median TTFT than Azure OpenAI. What explains the gap?Explain

A bar chart of median time to first token, Bedrock on-demand at 75 milliseconds and Azure OpenAI PTU at 50 milliseconds, with a question about why there is a 25 millisecond gap. — Figures from main.py, matching the lesson's Artificial Analysis numbers.

13. What do you accept in exchange for dedicated capacity on Bedrock Provisioned Throughput?Explain

A box for Bedrock Provisioned Throughput priced at 21 to 50 dollars per hour per unit, requiring a monthly commitment, with a question about what you accept in exchange for dedicated capacity. — Price and commitment terms from The Concept section.

14. Which summary of the FinOps surface across the three platforms is correct?Explain

Three equally styled columns, Bedrock, Vertex and Azure, each describing its native cost-attribution mechanism in neutral terms, with a question about which is cleanest, most flexible and most opaque. — Mechanisms from The Concept section, FinOps surface; no platform is highlighted.

15. Why does the Concept section recommend a two-provider minimum for product-critical LLM calls?Explain

A timeline where the lead model rotates quarter to quarter, Claude 3.7, then Gemini 2.5, then GPT-5, above two named incidents, the Azure OpenAI January 2025 incident and the AWS us-east-1 outage, with a question about why the lesson recommends a two-provider minimum. — Named events from The Concept section, Lock-in is the 2026 risk.

16. All three platforms meet the basic compliance checkbox. Where do they actually differ?Explain

Three rows, Bedrock, Azure OpenAI and Vertex AI, each listing its compliance basics with an identical checkmark, followed by a dashed box asking where they actually differ. — Checkmarks are uniform; nothing is ranked.

17. Your Azure OpenAI workload averages 30% sustained utilization of a PTU reservation. Based on the break-even band, what is the sensible choice?Apply

A 0 to 100 percent utilization axis with the lesson's 40 to 60 percent break-even band shaded and a single marker at 30 percent sustained utilization, with a question about the sensible choice at that point. — Band from The Concept section; only the 30% point is marked.

18. Your Azure OpenAI bill jumps 4x and you need to find which feature caused it. Why is this harder on Azure than on Bedrock with profiles?Analyze

Two bars for an Azure OpenAI bill, a short bar for last month and a much taller bar for this month, with no traffic change, and a question about which feature is costing more. — Illustrative bars; the lesson gives no specific dollar figures here.

19. A team on shared on-demand capacity sees a good median TTFT but P99 spikes in busy hours. Which idea from the lesson explains it?Analyze

A scatter of TTFT samples that sit low and flat for most of the run, labelled good median, with a handful of points spiking high near the end, labelled P99 spikes during busy hours, and a question about what explains them on shared on-demand capacity. — Illustrative distribution; no specific latency numbers given in the stem.

20. A manager rejects a second provider because it will double the LLM bill. What is wrong with that claim?Analyze

A bar chart with a typed primary spend bar and a manager's claimed bar at exactly double its height for adding a second provider, with a question about what is wrong with doubling the estimate this way. — Typed, illustrative figures, not the lesson's real redundancy numbers.

21. Your product needs Claude 3.7 Sonnet and GPT-4o and must survive a single-provider outage. Which design matches the Concept section?Design

Three equally styled catalog columns, Bedrock, Azure OpenAI and Vertex AI, with their real model families listed, under a requirement for both Claude 3.7 Sonnet and GPT-4o to survive a single-provider outage, and a question asking for the design. — Catalog membership from The Concept section.

22. A product on Azure OpenAI runs steadily near 80% PTU utilization and needs P99 TTFT under 100 ms. Which choice fits?Design

A 0 to 100 percent utilization axis with the lesson's 40 to 60 percent break-even band shaded and a single marker at 80 percent sustained utilization, with a question about which choice fits given a sub-100-millisecond P99 target. — Band from The Concept section; only the 80% point is marked.

23. On Bedrock, finance wants monthly cost per product feature with no post-processing. Which setup fits?Design

A request box on the Bedrock platform asking for monthly cost broken down per product feature with no manual post-processing, with a question about which setup fits. — Scenario from the lesson's FinOps surface section; the setup is the question.

Use It

code/main.py compares the three platforms on a synthetic workload. It models on-demand vs PTU economics, TTFT variance, and cost attribution fidelity. Run it to see where PTUs pay off and where the marketplace's model breadth outweighs a TTFT gap.

Check: Use It0/5

1. In the PTU sweep, one Azure PTU costs $10/hr and delivers 2.0M tokens/hr, with output priced at $10 per M tokens. What does the sweep print as the winner at 50% utilization?Apply

A table of four inputs, a 10 dollar per hour PTU rate, 2.0 million token per hour PTU capacity, a 10 dollar per million token output price and 50 percent utilization, with a question about which the sweep prints as the winner at that utilization. — Inputs match main.py's Azure PTU sweep.

2. In the first run (3.0M input and 1.0M output tokens per day), Vertex charges $1.25 per M input and $5.00 per M output. What does it print per day?Apply

A table of the first run's workload: 3.0 million input tokens, 1.0 million output tokens, and Vertex's prices of 1.25 dollars per million input tokens and 5.00 dollars per million output tokens, with a question about what Vertex prints per day. — Workload and prices from main.py's first run.

3. In the second run (30M input and 15M output tokens per day), what is Azure's on-demand cost at $2.50 per M input and $10.00 per M output?Apply

A table of the second run's workload: 30 million input tokens, 15 million output tokens, and Azure's on-demand prices of 2.50 dollars per million input tokens and 10.00 dollars per million output tokens, with a question about Azure's on-demand cost per day. — Workload and prices from main.py's second run.

4. With the Use It script's model of 3% gateway overhead plus 10% idle secondary headroom, what is the daily redundancy uplift on $50 of primary spend?Apply

A table of three inputs: 50 dollars of daily primary spend, 3 percent gateway overhead, and 10 percent idle secondary headroom, with a question about the total daily uplift from these two pieces. — Inputs from the lesson's Use It script.

5. In the second run PTU is enabled and Azure's dedicated median is 38 ms, yet Azure prints FAIL against the 100 ms P99 SLA. What happened?Analyze

A table for the second run with PTU enabled: 30 million input and 15 million output tokens a day, an Azure dedicated median TTFT of 38 milliseconds, an SLA of P99 under 100 milliseconds, and Azure's printed result of FAIL highlighted, with a question about why it fails despite the low dedicated median. — Values from the stem and a real run of main.py.

Ship It

This lesson produces outputs/skill-managed-platform-picker.md. Given a workload profile (models needed, TTFT SLA, daily volume, compliance requirements), it recommends a primary platform, a fallback, and a FinOps instrumentation plan.

Author's Note

I'd run main.py and read the PTU break-even sweep first. It holds the Azure rate fixed and steps utilization from 10% to 100%, so you see the exact row where the reservation starts to win. Prices and latencies in the toy are approximations that I'd treat as placeholders, and you should swap in the rate cards from Further Reading before trusting a number.

My bet is that the two-provider section is the one worth arguing with. The script prices redundancy as 3% gateway overhead plus 10% idle secondary headroom, which is 13% of primary spend on a 5M-token day. That is a small number next to a multi-hour outage, but it is not zero, and I'd rather you see it than take "negligible" on faith.

I'd also keep the comparator in proportion. It grades FinOps attribution with a letter and a label, and it can't tell you how much work Azure's tag instrumentation takes in your own org. The catalog question, which models you need, decides more than any row in its output. Start there, then use the numbers to break ties.

Exercises

Run code/main.py. At what sustained utilization does Azure PTU beat on-demand in the GPT-4o class sweep? Read the break-even row and compare to the advertised 40-60% band.
Your product needs Claude 3.7 Sonnet and GPT-4o. Design a two-provider deployment. Which model goes to which hyperscaler, what gateway sits in front, and what is the failover policy?
A regulated healthcare customer requires BAAs, US-East data residency, and sub-100ms P99 TTFT. Pick a platform and justify with three specific features.
You discover your Bedrock bill is up 4x this month with no traffic change. Without Application Inference Profiles, how would you find the culprit? With profiles, how long does it take?
Read the Azure OpenAI and Bedrock pricing pages. For a 100M-token/month Claude workload, which is cheaper: direct Anthropic API, Bedrock on-demand, or Bedrock Provisioned Throughput?

Key Terms

Term	What people say	What it actually means
Bedrock	"AWS LLM service"	Model marketplace across Claude, Llama, Titan, Mistral, Cohere
Azure OpenAI	"Azure's ChatGPT"	Exclusive OpenAI models in Azure datacenters with enterprise controls
Vertex AI	"Google's LLM"	Gemini-first platform with Model Garden for third-party models
PTU	"dedicated capacity"	Provisioned Throughput Unit, reserved inference GPUs, priced per hour
Application Inference Profile	"Bedrock tagging"	Per-product cost/usage profile with tags, CloudWatch-native
Model Garden	"Vertex catalog"	Vertex AI's third-party model section, separate from Gemini
Two-provider minimum	"LLM redundancy"	Policy of running every critical LLM path across ≥2 hyperscalers
BAA	"HIPAA paperwork"	Business Associate Agreement; required for PHI; provided by all three
Abuse monitoring	"the log watcher"	Provider-side safety scan on prompts/outputs; opt-out in enterprise