AI
Automatic Per-Prompt Model Selection
Azure OpenAI now carries five marquee model families on the standard tier: OpenAI, xAI Grok, Meta Llama, Mistral, and BFL Flux. With that breadth comes a real problem: most developers default to the flagship model for every call, then optimize later, if ever. Model Router is a trained classifier that sits in front of the pool and picks nano, mini, 4.1, or o4-mini for each prompt based on its complexity. No code changes are required. It appears as a single deployment in the model catalog and accepts the standard chat completions endpoint.
Benchmarking the Router Against GPT-4.1
Sethu ran the model router playground and 1,500-prompt benchmark (12:39) live on stage. A weather query went to nano. A nine-day Paris itinerary with a specific budget went to 4.1. Against 1,500 curated benchmark prompts, GPT-4.1 scored 83.1% accuracy. The router scored less than one percentage point lower.
“around 63% cost saving” — Sethu Raman
Azure Monitor shows the distribution: roughly 73% of prompts landed on 4.1-mini, 13% on o4-mini for harder reasoning, and the rest split between nano and 4.1. A developer building that logic by hand would need to run that evaluation themselves. Most would not.
Provisioned Throughput: Sizing, Reservations, and Flexibility
Standard tier is multi-tenant and carries only a reliability SLA. During peak hours, latency suffers. Provisioned throughput adds rate-limiter controls that guarantee each customer’s effective throughput and adds a latency SLA. The hourly rate runs $1-2 per PTU. A 50-PTU deployment costs roughly $100/hour at list price. A one-month reservation cuts that 40-50%; a one-year reservation cuts it up to 70%. The capacity calculator in the quota tab sizes workloads automatically: RAG plus basic chat plus summarization lands at about 50 PTUs. A new automatic exchange program lets customers swap region, deployment type, or term without a support ticket.
Prompt Caching and Dynamic Spillover
Prompt caching fires automatically for any prompt of 124 tokens or longer. On the standard tier, cache hits cost 50-75% less. On provisioned throughput, they are free. Saloni ran the prompt caching live load test (36:32) at 100% PTU utilization with 15,000-token inputs. Throughput climbed from 3.7 million to 6.8 million tokens per minute once the cache warmed. Manus went from a 30% to 70% cache hit rate after Microsoft tuned the implementation. Dynamic spillover routes overflow to the standard tier automatically when PTU capacity is saturated, avoiding 429 errors.
“That’s one of the biggest secrets, I would say,” — Steve Sweetman, on cached tokens being free under provisioned throughput
Batch API at Scale: 75 Billion Tokens in Three Days
CVS Health receives millions of faxes, scans, and clinical notes daily. A physician seeing 12-15 patients cannot read it all. Their clinical insights engine searches each document for specific values (ejection fractions between 35-45%, bronchiectasis in a cardiology note) and flags them before the visit. The scaling problem: 17 million requests, 75 billion tokens, 100 days and over $300,000 on standard pay-as-you-go. Four batch features cut through it: dynamic quota beyond the 10-billion-token cap, a 10,000-file upload limit (up from 500), bring-your-own-blob, and a while-loop queuing construct with fail-fast feedback. Half a day to queue. Three days total.
“75 billion tokens in just under three days,” — Christopher Schubert
Notable Quotes
around 63% cost saving Sethu Raman · ▶ 16:47
That’s one of the biggest secrets, I would say, Steve Sweetman · ▶ 33:57
75 billion tokens in just under three days, Christopher Schubert · ▶ 55:46
Key Takeaways
- Model Router picks nano, mini, or o4-mini per prompt, saving up to 63% over always using GPT-4.1.
- Provisioned throughput reservations cut the hourly PTU rate by up to 70%; buy capacity before the reservation.
- Batch API’s dynamic quota and 10,000-file limit let CVS Health process 75 billion tokens in three days for just over $100,000.