AI
Migrating 680 Intents to Copilot Studio
Commonwealth Bank of Australia runs a chatbot with 680 intents across iOS, Android, and web for over 10 million customers. When CBA migrated to Copilot Studio, it ran programmatic conversion of legacy intents in parallel with manual review, then layered role-based access control, firewall policies, and data-integrity checks on top. The full team, roughly 30 engineers, conversation designers, and product owners, completed the one-to-one topic mapping in five months with zero defects.
Connecting a Regulated Bank’s APIs to a SaaS Agent
Every message enters CBA’s infrastructure before reaching Copilot Studio via DirectLine. Responses needing bank data or custom guardrails (tone checks, financial-advice compliance) cross back through a firewall. When the project started, custom connectors couldn’t handle MTLS or payload signing, so the team routed calls through Power Automate instead. That workaround came with a hard timeout, a restricted set of C# namespaces, and cold-start risk on the underlying Azure Function. CBA solved all three, and has since migrated most of that path to VNet and OAuth via Azure API Management.
Keeping Makers Out of Git Without Losing Auditability
CBA’s conversation designers are not engineers, and the team didn’t want to upskill them into Git. The solution was a custom Power App CI/CD demo (19:15): a Power App interface that lets makers pick a solution, choose a target environment, and commit or pull changes. GitHub Actions and Linux runners handle everything underneath. The bank maintains roughly 15 environments across separate non-prod and prod tenants, and no human has direct access to the prod tenant. All prod deployments run automated end-to-end through GitHub pipelines.
Monitoring a 10-Million-User Agent in Production
Copilot Studio’s built-in analytics weren’t enough for a bank that already operates enterprise-wide monitoring. The team connected the agent to Application Insights and wrote custom Kusto queries that measure how long each node in a topic takes to execute. One query showed the first bot message back in 16 milliseconds and the next call at 1.8 seconds, pinpointing exactly which node to investigate. Conversation transcripts also export into CBA’s enterprise data warehouse for postmortems and conversation analysis.
Platform Roadmap: Source Control, MCP, Multi-Agent, and BYOM
Bas Brekelmans walked through four features shipping at or around Build. Native source control integration (now generally available) lets makers commit directly from the Copilot Studio UI. MCP server support adds a discovery layer so APIs built for app-to-app calls expose agent-readable descriptions without tight coupling. Multi-agent orchestration routes sub-tasks to smaller specialized agents when a single context with hundreds of tools starts losing accuracy. Bring-your-own model from Azure Foundry lets teams pin response generation to a fine-tuned model, with a few-week preview window before Microsoft rolls updates out broadly.
Q&A
After this project, would CBA continue with Copilot Studio or shift to the Agents SDK for new agents? CBA hasn’t decided yet which new agents will use Copilot Studio versus the Agents SDK, but the extendability of the SDK means they can bring existing agents into the platform without tight coupling to Microsoft. ▶ 46:37
Can Azure AI Foundry evaluation tools test Copilot Studio agents, and how do you balance pre-release testing against post-release monitoring for content harm and prompt injection? Brekelmans confirmed Application Insights provides detailed per-turn telemetry, a native RAG and Q&A test suite is launching as an early preview, and a Microsoft Australia team is near-shipping Foundry evaluator integration as an open-source tool. ▶ 49:18
How does Copilot Studio handle token quotas and horizontal scaling when many users hit the model simultaneously? Brekelmans said GPU capacity and demand forecasting are managed entirely by the Copilot Studio engineering team; customers only need to ensure they have sufficient credits. ▶ 52:42
Why did CBA migrate from the legacy chatbot, how long did it take, and does the chatbot execute transactions or only guide customers? Verma said migration took five months, the chatbot handles full workflows like raising disputes but directs customers to the app for transactions like balance transfers to avoid duplicating effort. ▶ 53:47
Key Takeaways
- CBA migrated 680 chatbot intents to Copilot Studio in five months with zero defects using a hybrid programmatic-plus-manual approach.
- Enterprise teams can wrap Copilot Studio in standard SDLC tooling: GitHub Actions, shift-left testing, and cross-tenant deployment pipelines.
- Copilot Studio’s ‘no cliffs’ design lets teams mix low-code authoring with custom code, MCP servers, and bring-your-own models as requirements grow.