Autonomous Agents vs. the Rest of the Spectrum

▶ Watch (06:24)

Over 230,000 organizations have adopted Copilot Studio. In the last three months alone, they created more than one million agents, covering over 90% of the Fortune 500. But Smith draws a firm line between the spectrum of agents. A basic RAG lookup is not an autonomous agent. The distinction: autonomous agents respond to events (an email arrives, a database row changes, a schedule fires) without a human initiating the conversation. They reason, plan, and act. Building one requires stronger orchestration, observability, and governance from day one.

Ten Customer Lessons That Shaped the Roadmap

▶ Watch (11:49)

Cram 20 documents, 10 connectors, and an entire sales department into one agent and it fails. Smith’s fix: build specialist agents, one task each, then wire them together.

“So building these SME’s is key to this outcome, to reliability.” — Ray Smith

That insight drove multi-agent orchestration, released at Build. Nine more customer lessons produced nine more features: o1/o3 reasoning for contract and M&A analysis, MCP servers to wrap 40-parameter APIs, agent flows for deterministic sub-tasks, activity streams for debuggability, and centralized governance so IT can cap capacity and audit every run.

Suncorp’s RAG Deployment: Two Million Interactions, 50% Fewer Escalations

▶ Watch (34:19)

Suncorp fields two million customer contact center interactions every quarter. Before RAG, staff typed keywords into a search box, got a list of ten articles, and guessed which one answered the caller’s question (all while the customer waited on hold). Shen’s team replaced that with a vector search plus Azure OpenAI pipeline that returns a cited answer in seconds. Pilot results: 50% drop in escalations to next-level support. A separate single-view-of-claims tool has since processed over three billion input tokens and one billion output tokens.

Designing for Production: Governance Tradeoffs in Agentic Systems

▶ Watch (43:09)

Full autonomy is not the right answer for every step. Shen frames the core tradeoff: a deterministic flow gives you confidence every required step runs, but it hits walls when something unexpected arrives. A planning agent handles more edge cases but introduces variability. Most production systems need both: a master agent watching the end-to-end process, specialist sub-agents for defined tasks, and an explicit exception path back to a human or a dedicated exception-handling AI. Auditing is not optional. Regulators will ask which data drove each decision, and the answer needs to exist in logs.

Live Demo: Multi-Agent Claims Processing End to End

▶ Watch (47:39)

Smith ran a four-agent claims pipeline live. A Claims Manager Agent receives the trigger and calls three sub-agents in sequence: a Policy Researcher (o3 deep reasoning over the policy document and police report), an AU10TIX Validation Agent (license, registration, and repair shop ID checks via partner API), and a Decision Analyst. The Comms Agent then either emails an approval directly to the customer or fires a Teams adaptive card to a human adjuster. The auto claims manager live run (52:37) shows the full activity stream, drillable to each tool call and chain-of-thought prompt.

Notable Quotes

So garbage in is garbage out. Ray Smith · ▶ 12:41

So building these SME’s is key to this outcome, to reliability. Ray Smith · ▶ 14:13

this application has processed over three billion input tokens Jon Shen · ▶ 38:05

Key Takeaways

  • Multi-agent systems beat single uber-agents: focused SME agents per task deliver reliable, debuggable outcomes.
  • Email was 47% of agent triggers six months ago; custom API connections have cut that to 17% as maturity grows.
  • Governance tooling (activity streams, analytics, capacity caps) is now the top ask from production-scale customers.