Choosing the right model from 10,000 options

▶ Watch (06:02)

Azure AI Foundry’s catalog lists over 10,000 models through a Hugging Face partnership. Leaderboards rank them on quality, cost, and throughput. Sending a geography question to a reasoning model wastes compute; model router routes each query to the smallest capable model automatically.

“Model-router takes the selection toil from your head.” — Yina Arenas

The model router live demo (09:38) shows nano handling a capital city question and a reasoning model taking a complex planning query. Switching from GPT-4.1 to model router cuts costs up to 60%, with a 1-2 point accuracy trade-off. The Foundry local biography demo (12:22) loads Phi-4-mini onto a local GPU; the GPU memory spike is visible on screen.

What an agent actually is, and when to use one

▶ Watch (14:34)

Process automation has existed for decades. The difference now is an LLM driving the control flow, deciding which tools to invoke, maintaining memory across turns, and accepting multimodal inputs. Arenas recommends building small, focused agents first, validating each like a unit test, then wiring them together.

The Foundry Agent Service went GA at Build. It brings enterprise features: bring-your-own storage, network-aware deployment, authentication token pass-through, and built-in integrations for Fabric, SharePoint, Bing grounding, and Azure AI Search. It supports A2A, MCP, LangChain, CrewAI, and the OpenAI Assistants and Responses APIs, so agents built outside Foundry can connect without a rewrite.

Building a multi-agent workflow from real code

▶ Watch (20:04)

Processing Hanselman’s 1,000 episodes with Whisper cost about $100. Fine-tuning a nano model on that corpus cuts per-run cost for the next 1,000 to $1.50.

“But that is really bringing a sledgehammer to a scalpel fight.” — Scott Hanselman

The show notes agent demo (21:32) pastes a transcript into the fine-tuned nano model and returns takeaways, notable quotes, and a summary. Semantic Kernel and AutoGen, now converging on a shared runtime, connect the individual agents into a pipeline: content agent, link verifier, summarizer. The multi-agent pipeline live run (36:08) kicks off a dotnet process and outputs a markdown show notes file with a verified link table in one pass.

“Everything we built here is legit.” — Scott Hanselman

Observability, evaluation, and security for agent apps

▶ Watch (36:51)

Foundry’s observability layer runs on OpenTelemetry, so any OTel-compatible dashboard (Grafana, .NET Aspire, Azure Monitor) can display agent threads and per-turn metadata. Each thread stores conversation memory; each run logs inputs, outputs, and metadata for every call.

Evaluations plug into GitHub Actions, so each push to production can trigger quality, safety, and intent-resolution checks. At Build, Microsoft added task adherence and indirect jailbreak detection as new eval types. On security, each agent gets its own Entra identity with scoped permissions. Data flows to Purview for classification; Defender integrates at the platform level.

Notable Quotes

Model-router takes the selection toil from your head. Yina Arenas · ▶ 9:42

But that is really bringing a sledgehammer to a scalpel fight. Scott Hanselman · ▶ 20:44

Everything we built here is legit. Scott Hanselman · ▶ 44:33

Key Takeaways

  • Model router cuts costs up to 60% versus GPT-4.1 with only a 1-2 point accuracy trade-off.
  • Fine-tuning Hanselman’s 1,000-episode corpus dropped per-run cost from $100 to $1.50.
  • Foundry Agent Service GA assigns agents their own Entra identity for scoped, auditable permissions.