AI
Azure AI Speech Moves Into AI Foundry
Azure AI Speech has moved entirely into AI Foundry, replacing Speech Studio. Dong Li announced three new products at Build: a Video Translation API that dubs content into multiple languages in one call, a Voice Live API for full speech-to-speech agents, and expanded TTS avatar support.
The Pacers Sports and Entertainment case study showed what domain-tuned speech models can deliver. Trained on hundreds of hours of Pacers and Fever game calls, their model reported:
“we boosted accuracy from 91% to nearly 99%,”
with response times dropping from seconds to milliseconds.
Video Translation: 70 Languages, One API Call
Video Translation preserves the original speaker’s emotion and tone, syncs audio to lip movements, and covers more than 70 languages. Since its Ignite November 2024 preview, over 700 customers have adopted it. LinkedIn uses it to translate 23,000-plus learning courses worldwide.
Two integration paths exist: a REST API that auto-detects speakers and returns dubbed video, and a Semantic Kernel plugin for embedding translation into agents. A Foundry Playground lets developers test configurations before calling the API.
Voice Live API: One WebSocket for Full Speech-to-Speech Agents
Building production voice agents means wiring together STT models, TTS engines, audio libraries, and a generative AI model, each with its own latency cost. Voice Live API collapses all of that into one WebSocket connection, announced in public preview at Build 2025.
The API bundles Azure STT and TTS, model choice (GPT-4o, 4o-mini, or Microsoft Phi), plus server-side noise suppression, echo cancellation, and semantic end-of-turn detection. Developers can start fresh or add voice-in and voice-out to an existing text-based agent without re-architecting their stack.
Gainsight Demo: Voice Agents Serving the Long Tail of SaaS Customers
The long tail of SaaS customers is about 80% by volume, 20% of revenue (roughly $50 billion out of $300 billion annual SaaS spend). Those customers get no white-glove service because human CSMs are too expensive to scale. Prem Parameswaran from Gainsight built two Voice Live API demos overnight to show agents can fill that gap.
In the Gainsight renewal-desk voice agent live demo (19:55), the audience supplied company and license details live; the agent used them correctly.
“With agents, that’s actually possible.”
Voice Customization and the Developer API
The Voice Live API developer model is built on top of the existing real-time WebSocket interface. Switching AI models (GPT-4o, 4o-mini, or Phi) is one parameter in the connection string. Adding a Phrase List for domain terms, or enabling server-side noise suppression, is a few lines of config. Custom neural HD voices are now in public preview, announced at Build.
In the Hasya custom-voice demo in AI Foundry Playground (34:23), about 300 recorded sentences trained a voice clone. The deployed agent answered in Hasya’s own voice while calling itself Andrew, confirming the voice and persona are independently configurable.
Who Is Already Running This in Production
CommerzBank, a German bank with over 400 branches, first showed its avatar project at Microsoft Build in November 2023. Gerald Erll confirmed it is now live. The avatar runs on GPT-4o and handles general banking questions, customer-specific requests like interest rate lookups, and first transactions such as blocking or unblocking a credit card.
“which is capable to do transactions.”
The Government of Malta trained custom avatars using the same Voice Live API, providing multilingual public services to citizens. Anchor demonstrated earbuds powered by the API for hands-free meeting summaries and navigation.
Notable Quotes
With agents, that’s actually possible. Prem Parameswaran · ▶ 23:53
we boosted accuracy from 91% to nearly 99%, Pacers Sports & Entertainment · ▶ 4:23
which is capable to do transactions. Gerald Erll · ▶ 53:36
Key Takeaways
- Voice Live API unifies STT, TTS, and GenAI model selection in a single WebSocket connection, replacing bespoke audio pipelines.
- Video Translation reaches 70-plus languages with voice cloning and lip sync; over 700 customers adopted it since Ignite November 2024.
- Custom neural HD voices now in public preview, trained from roughly 300 audio samples and deployable via a single endpoint ID parameter.