Continuously improve your Agent in production

by Ami Patel, Han Che

Day 3 · Demo Session · 15 min · 2 min read

Applied AI AI Infrastructure

TL;DR

Amy and Han demo Foundry Observability, a new Azure AI Foundry feature that streams per-trace quality scores to Application Insights using two agent-specific evaluators: task adherence and intent resolution. They show how to filter production traces scoring below 3, diagnose a vague return-policy response that scored 1 and 2 respectively, then wire evaluation into Azure DevOps via a CI/CD extension that benchmarks candidate agents across 15 test queries before any code ships. Replacing the original agent with a Bing-search variant pushed all quality metrics back above baseline within one deployment.

Tracing and Continuous Evaluation in Production

▶ Watch (01:30)

Foundry Observability wires your project client to Application Insights through Azure Monitor and streams a quality score for every production conversation. Two agent-specific evaluators ship with it: task adherence (does the agent stay on its job?) and intent resolution (does it actually answer the question?). You can sample 100% of traffic or dial it down. The reasoning field explains each score, though it defaults to off for sensitive agents.

When scores dip below 3, you filter to those traces in the filtering low-scoring traces in Foundry (05:16). A return-policy query came back vague and scored task adherence 1, intent resolution 2.

CI/CD Pipeline Evaluation with Azure DevOps

▶ Watch (06:32)

A new Azure DevOps marketplace extension, AI Agent Evaluation, triggers an evaluation run on every commit or PR. It needs four inputs: the AI Foundry project endpoint, a judge model (Han picks GPT-4o), one or more agent IDs, and a test dataset. The first agent ID is the baseline; every other is compared against it.

The Azure DevOps AI Agent Evaluation pipeline run (09:28) takes 10-15 minutes. Results include per-query scores, latency, and token cost. Color-coded statistical significance, light green for marginal and dark green for strong, makes it quick to spot the better candidate.

Closing the Feedback Loop After Deployment

▶ Watch (12:49)

Deploying the Bing-search candidate closed the feedback loop. The production score recovery dashboard (13:05) shows intent resolution, relevance, and task adherence all climbing after the swap. Safety scores, which had been unsteady, held at 0. Token consumption rose with traffic, an expected trade-off for a more capable agent.

The Foundry monitoring view connects directly to Azure Monitor Application Insights. You can save the dashboard, share it with your team, and set Azure Alerts on any quality metric, all from the same view.

Notable Quotes

to measure agent quality, task adherence, and intent resolution. Amy · ▶ 2:39

It’s kind of saying like it could be 30 days, Amy · ▶ 5:44

actually introduced statistic analysis into this and we also provide Han · ▶ 11:57

Key Takeaways

Foundry Observability streams per-trace quality scores to Application Insights so you can filter bad responses by score threshold.
The Azure DevOps AI Agent Evaluation extension compares multiple candidate agents against a baseline using 15+ test queries before any code merges.
Deploying the higher-scoring Bing-search candidate pushed intent resolution, relevance, and task adherence scores visibly above the original agent.

Filed under

Applied AI AI Infrastructure