Choosing and Deploying Models

▶ Watch (03:08)

Model Router analyzes each incoming prompt and routes it to a simpler or more complex model based on complexity. A question like “What is the weather in May in Paris?” lands on GPT-4o; a harder query goes to a reasoning model like o1. The Model Catalog covers providers from Azure OpenAI to Mistral, Meta, NVIDIA, and the newly added Grok 3. The Model Leaderboard plots quality against cost and throughput so you can compare candidates on the dimensions that matter before committing to a deployment.

Building Agents in VS Code

▶ Watch (06:20)

The VS Code Azure AI Foundry extension lets you browse models and spin up an agent without leaving the editor. Paul created an invoice agent using GPT-4o with a short system prompt, then added the Code Interpreter, one of several built-in tools that run Python in an isolated sandbox. In the Code Interpreter generating a PNG invoice (11:38), the agent wrote Python, executed it, and returned a downloadable PNG file. Foundry surfaces the generated code in the UI so you can inspect exactly what ran.

Connecting Agents to Structured and Unstructured Data

▶ Watch (13:04)

The Foundry SDK’s project client connects to a project with one endpoint and surfaces inferencing, evaluation, and observability from the same handle. Paul used GitHub Copilot in agent mode to generate field-extraction regex for invoice number, date, and customer name. A script ran five invoices through the agent, parsed the output, and wrote structured records to Cosmos DB and PNG files to disk. In the Azure MCP server querying Cosmos DB from Copilot (19:43), Copilot queried the container in natural language with no portal visit. Switching from GPT-4o to DeepSeek took one line.

“That’s the only line of code I have to change.” — Paul Shealy

Tracing and Observability

▶ Watch (29:19)

Enabling tracing takes two lines: one to get the App Insights connection string from the project client, one to configure writing to Azure Monitor. After that, every agent step (including tool calls and Cosmos DB queries) appears in the Foundry portal’s Tracing tab as an OpenTelemetry span. Paul’s demo showed the full span tree for the invoice agent, including the get_overdue_invoices tool call with its returned JSON list of customer names and due dates. Foundry writes spans in the OpenTelemetry format, so any tool that reads OTel traces can consume them.

Evaluation and CI/CD Integration

▶ Watch (34:02)

Foundry’s evaluators cover quality (coherence, fluency, relevance, groundedness) and safety (jailbreaks, hate speech, violence, self-harm). For agents, intent resolution, task adherence, tool call accuracy, and response completeness each return a scored JSON object with LLM-generated reasoning. During the tool call accuracy miss exposing a data gap (42:17), the evaluator scored 0.25 out of 1 and the trace revealed the bug: the agent passed a customer name instead of an email address because the invoices never stored email fields. The GitHub Actions CI/CD eval run (43:31) shows how to gate every deploy on those same scores.

“Tool call accuracy, we got a 0.25.” — Paul Shealy

Notable Quotes

One of our new announcements is the Model Router, Yina Arenas · ▶ 3:08

That’s the only line of code I have to change. Paul Shealy · ▶ 22:58

Tool call accuracy, we got a 0.25. Paul Shealy · ▶ 42:17

Key Takeaways

  • Model Router selects the right model per prompt automatically, removing manual model selection from your workflow.
  • Two lines of code connect an Azure AI Foundry project to App Insights for full OpenTelemetry trace capture.
  • The GitHub Actions agent evaluator catches regressions on intent resolution and tool call accuracy before every deploy.