AI
Why Distributed Tracing Beats Logs for Root Cause Analysis
Distributed tracing captures the full lifespan of a request across every service it touches. Logs give you a timestamp; a span gives you start and end, so you get duration. Logs carry no enforced structure, so two services can record the same event with different field names. Tracing enforces semantic conventions and correlation IDs across the call graph. That consistency is what makes root cause analysis possible in large systems. When latency spikes on an API, tracing shows exactly which downstream service introduced it.
The Jaeger Data Model: Traces, Spans, and Tags
A trace represents one end-to-end request. It is composed of spans, where each span covers a single operation such as a database call or an HTTP request. Private method invocations are skipped. Spans carry tags (structured attributes) that record the URL, HTTP method, and status code for a web call, or the full SQL query for a database call. Jaeger relies on OpenTelemetry data semantics for this model. The project no longer ships its own instrumentation SDK; it points users to the OpenTelemetry SDK instead.
Live Demo: Reading a Trace and Spotting Performance Problems
The public demo at jaegertracing.io runs HotROD, a microservice app that simulates a ride-hailing service with four services (frontend, customer, driver, routes) and two databases (MySQL and Redis). Redis is called roughly ten times more often than MySQL. In the trace timeline, the first span took 600 milliseconds and the MySQL call 255 milliseconds. A staircase pattern in the timeline flags sequential calls that could be parallelized. Trace diff lets you compare two traces side by side and highlights spans that appear in one but not the other.
Deriving RED Metrics from Spans with Prometheus
Because every span has a start and end time, Jaeger can extract latency, error rate, and request rate from the span stream and write them to Prometheus. The monitor tab in the Jaeger UI surfaces those three metrics per service and per endpoint. You can push data via Prometheus remote write, Prometheus scraping, or OTLP HTTP. Querying runs through Prometheus; it is the only metrics backend Jaeger supports today. This brings Jaeger closer to a traditional APM tool without requiring a separate metrics pipeline.
Jaeger V2: One Binary, OpenTelemetry Collector Internals
Jaeger V2 reimplements all Jaeger functionality as OpenTelemetry Collector components. There is a Jaeger storage extension, a Jaeger storage exporter, and a Jaeger UI extension. Internally, Jaeger now uses the PData pipeline data structure from the collector. The old V1 multi-binary model (separate collector, agent, and query binaries) is gone. One binary handles everything, configured with a standard OpenTelemetry Collector YAML file. V1 was removed from the codebase entirely in January 2025. ClickHouse, previously a plug-in, is now built in.
Jaeger MCP Server: Progressive Disclosure for LLM Queries
The MCP server lets an LLM query Jaeger traces without dumping an entire trace into the context window at once. The first call returns trace summaries. Follow-up calls fetch the critical path, errors, or individual span details for spans worth investigating. That layered approach keeps token usage low on large traces. The roadmap adds an AI assistant directly in the Jaeger UI and observability support for generative AI workloads, including token counts and prompt data. Both features are open for collaboration on GitHub.
Q&A
Does Jaeger handle storage retention? Each storage backend has its own mechanism. Elasticsearch supports index rollover; Cassandra supports TTL on tables; in-memory storage lets you cap the number of traces kept. ▶ 27:46
How do you model Kubernetes controller reconciliation loops in traces? Use span links to connect three separate traces and jump between them in the Jaeger UI. ▶ 25:48
Is there a Jaeger data source for Perses dashboards? Not yet from the Jaeger side, but a community member offered to implement it. Prometheus metrics from Jaeger are already queryable in Perses as standard Prometheus metrics. ▶ 23:53
Notable Quotes
in my opinion, why people use tracing is actually the root cause analysis. Since we see everything what is happening in the system, we can reason why it happened and why it failed. Pavol Loffay · ▶ 03:00
Tracing is essentially like logging on steroids with three major differences. Pavol Loffay · ▶ 05:44
it doesn’t return you the whole trace at once because that can feed the fill in the context window so quickly. trace can have so many data. Pavol Loffay · ▶ 17:58
everyone has to do some some AI stuff, but no, I think this is really cool Pavol Loffay · ▶ 17:40
Key Takeaways
- Jaeger V2 ships as a single binary configured with a standard OpenTelemetry Collector YAML file.
- The MCP server fetches trace summaries first, then span details on demand, keeping LLM token use low.
- Jaeger derives latency, error rate, and request rate from spans and pushes them to Prometheus automatically.
About the Speaker(s)
Pavol Loffay is a principal software engineer at Red Hat working on open-source observability technology for modern cloud-native applications. He contributes to and maintains the CNCF projects OpenTelemetry and Jaeger, and has been involved with the Jaeger project since its inception roughly ten years ago.