AI
Why Agents Need Observability Built In
Agents misbehave in production for the same reason they misbehave in demos:
“agents are non-deterministic. That’s not just a [problem for demos].” — Amy Boyd
That non-determinism has three practical consequences. Reliability requires evaluating quality and safety during the build phase, not afterward. It requires monitoring agents continuously as models, customers, and environments change. And it requires an optimize step that turns observability data into concrete fixes, not just dashboards. All three need to be in place before the first customer request arrives.
Building a Traceable Agent in the Portal
Starting in the Foundry portal keeps early decisions manageable. At ai.azure.com, a new project deploys GPT-4o by default, and within minutes you have a named agent with system instructions, a Bing web search tool, and a playground to test prompts. Tracing is off until you connect App Insights, a one-click step in the agent’s Traces tab. Once connected, every conversation returns estimated token cost alongside AI quality and safety scores. Boyd walked through the Foundry portal agent demo (23:11), showing a Contoso Travel agent that flagged low task-adherence on the first real query.
From Portal to Code: SDK Notebooks and Workflow Agents
Five Jupyter notebooks run in a GitHub Codespace with all dependencies pre-installed. They build from a bare prompt agent through function tools (car, flight, hotel search) to a workflow agent, where a concierge orchestrates three specialist sub-agents via a declarative YAML definition. OTEL tracing can write to the local console or push to Azure Monitor. Narasimhan showed the workflow agent multi-agent trace (46:23), where per-agent token costs appear in a single trace view and the underperforming agent is immediately identifiable.
Evaluation and Red Teaming in Code
The evaluations notebook runs quality, safety, and agentic evaluators in batch against a generated test dataset, checking intent resolution, tool-call accuracy, and task adherence. One failure appeared when the agent cited August 2024 flight data instead of 2025, flagged as a groundedness failure with a plain-language explanation. Red teaming goes further:
“a second AI to attack your first AI.” — Nitya Narasimhan
The red teaming agent takes a risk category list and generates adversarial prompts, including l33t-speak substitution and crescendo attacks, then reports which guardrails were bypassed.
The Observe Skill: Coding Agent as Eval Loop
The Observe skill (early preview, released two weeks before the talk) turns the entire observability loop into a conversation. Invoked through GitHub Copilot Chat using the Foundry MCP server, it inspects the agent’s instructions, generates a synthetic evaluation dataset, runs a batch eval, reports failures, optimizes the system prompt, and re-evaluates automatically. Narasimhan ran the Observe skill live (65:23) on a bare Contoso Travel agent. The skill reached version 10 across multiple iterations, identified version 5 as best, and rolled back. A human-in-the-loop checkpoint at each step prevents unchecked prompt drift.
Notable Quotes
agents are non-deterministic. That’s not just a Amy Boyd · ▶ 5:16
the notion of observability it’s not enough Nitya Narasimhan · ▶ 34:42
a second AI to attack your first AI. Nitya Narasimhan · ▶ 59:03
Key Takeaways
- Build traces and evaluators into agents from day one, not as a post-production add-on.
- Trace-linked evaluations cut time from failure detection to root-cause diagnosis.
- The Observe skill automates the eval-optimize loop so developers can guide rather than manually script each step.