AI
Why Distributed Agents Break Traditional Observability
Monoliths gave way to microservices, microservices went event-driven, and now agents sit inside those same distributed systems making nondeterministic decisions. Traditional tracing assumed a request-response flow. Agents do not work that way. The number of spans varies between runs. Reasoning lives outside the running process, delegated to an LLM. Context propagation breaks across the mix of frameworks, SDKs, languages, and protocols a single agentic application can involve. That combination makes the standard playbook unreliable.
The Pizza Store Demo: 10 Services, 4 Agents, One Order
The demo application takes a pizza and beer order through a store manager agent that orchestrates three sub-agents: drinks, cooking, and delivery. Those agents run in separate containers and talk over the A2A protocol. MCP connects the LLMs behind each agent to Go microservices that track inventory, ovens, and bikes. Ordering a pepperoni pizza and a beer produced between 275 and 300 spans in a single trace. The cooking agent waits on a long-running oven reservation before the delivery agent picks up the order.
Dapr Workflows: Durability Across the Agent Chain
Dapr workflows give the multi-agent orchestration an audit trail and a resume point. If the cooking agent crashes mid-execution, the order restarts from the cooking step rather than from the beginning, avoiding a full replay of all LLM calls. Per-agent durability covers the tool and LLM calls inside a single agent. Workflow-level durability covers the entire order from placement to delivery. Agents are written in Java using LangChain4J. Dapr wraps that framework transparently to generate durable workflow state without changing the agent code.
OpenTelemetry Instrumentation Across Java, Go, and a Bash Script
The OpenTelemetry operator auto-instruments the Java services via the Java agent and the Go services via an eBPF sidecar. Each Go pod showed two containers: the application and the auto-instrumentation container. Dapr’s sidecar also emits traces out of the box. For the bike-service skill, which ran a bash script, auto-instrumentation could not reach inside the process. The fix was manual: extract the W3C trace parent, write it into an environment variable for the script’s execution environment, then propagate it with a curl call. It worked, but it required filing upstream PRs to patch broken context propagation in the A2A protocol.
Reading 275 Spans: GenAI Semantic Conventions and What Is Still Missing
A single pizza order generated 275 spans across 10 services, visible in Jaeger. GenAI semantic conventions expose the model name, system prompt, completion text, and token counts per LLM call. That data is there, but finding the reasoning steps inside 275 spans is slow. The number of spans is not fixed between runs because agent loops are nondeterministic, so span-count assertions in tests no longer work. The speakers noted that 275 spans for one pizza and one beer is too many, and that agentic frameworks still need to standardize how they surface agent-level information on top of raw GenAI spans.
Lessons Filed Upstream and What Comes Next
Building this demo produced multiple pull requests against the A2A protocol and related libraries to fix context propagation gaps. In Java, the recommendation is to avoid auto-instrumentation when using frameworks like LangChain4J because the Java agent can clash with framework internals. Skills as a concept were introduced in December 2025 and have no OpenTelemetry semantic conventions yet. Kasper Nissen is drafting an OpenTelemetry maturity support framework, tracked as a community issue, to score how well individual projects support OpenTelemetry. The goal is to let teams pick a framework knowing its observability story before they commit.
Notable Quotes
49% of all the respondents in in that survey are running open telemetry in production these days and an an additional 26% are evaluating right now. Kasper Borg Nissen · ▶ 11:44
There’s 300 spans in this. Kasper Borg Nissen · ▶ 21:02
more telemetry is not always equal to to better telemetry, right? Because 275 spans is is way too much for for ordering a single pizza and a and a beer in this case. Kasper Borg Nissen · ▶ 24:27
And that highlights that auto instrumentation will take you that far, right? Like when you have these kind of interactions that are a little bit off script or in this case in a bash script you would just need to actually go and and do it manually Mauricio Salatino · ▶ 24:03
Brace yourself with skills. Mauricio Salatino · ▶ 28:34
Key Takeaways
- A single pizza order across 4 distributed agents produced 275 to 300 spans.
- Dapr workflows let a crashed cooking agent resume mid-order rather than restart all LLM calls.
- W3C trace parent propagation through bash-script skills requires manual instrumentation and upstream PRs.
About the Speaker(s)
Kasper Borg Nissen is a Principal Developer Advocate at Dash0, a CNCF Ambassador, former KubeCon+CloudNativeCon Co-Chair, Golden Kubestronaut, KCD Organizer, and CNCG Group Organizer. He co-founded Cloud Native Nordics to unite meetups across the region and helps make observability practical for developers.
Mauricio Salatino, known as salaboy, is an Open Source and Ecosystem Engineer at Diagrid, where he works on the Dapr project. He is a Java Champion, Cloud Native Ambassador, and international keynote speaker with ten years of experience building tools for cloud-native developers.