AI
Why Wasteful Telemetry Gets Created in the First Place
Auto instrumentation dropped into a Dockerfile floods a telemetry backend with data that is expensive to store and hard to interpret. Delivery Hero compounded this with a federated structure inherited from acquisitions. Subsidiaries that ran their own businesses set their own observability conventions, producing telemetry with no common schema. The OpenTelemetry project itself carried responsibility too. The “instrument all the things” philosophy from the OpenTracing era left teams sitting on metrics they have never queried, afraid to delete anything.
Real Examples: PII Leaks, Traces of Doom, and Bad Service Names
Over 700 customer phone numbers landed in Delivery Hero’s observability backend after a database library logged a uniqueness constraint error that contained the phone numbers. A separate trace ran for 7 days and 12 hours, generated by auto instrumentation attached to a pub-sub consumer. Service name fields contained broadcast IP addresses, pod specs from the previous deployment, AWS S3 bucket identifiers, and Java SDK strings. Vendor-specific attributes from a vendor the team stopped using years ago were still shipping with every span.
The Holy Trinity of Bad Telemetry
Kröhling grouped the problems into three categories: volume, governance, and sensitive data. Volume means too many logs, traces, spans, and metrics with no signal value, such as a health check trace emitted every five seconds. Governance means inconsistent attribute naming, where the same concept appears as “service,” “app,” and “application” across teams. Sensitive data means PII ending up in a backend that was never designed to hold it. All three increase cost and reduce the usefulness of every query.
Collector-Level Controls: Transform, Delete, and Deduplicate
The OpenTelemetry Collector’s transform processor can mask PII with a regular expression or delete all attributes whose names match a specific vendor string, without touching application source code. The log dedup processor groups similar log lines over a configurable interval. Instead of emitting 1,000 logs in 10 seconds, it emits one log with a counter attribute recording how many matching events occurred. These fixes work at the pipeline layer and buy time while source-level changes are planned.
Source-Level Governance: Weaver and Agentic PR Review
OpenTelemetry Weaver lets teams define a schema manifest that declares metric names, instruments, units, required attributes, and allowed values. Weaver then compares actual emitted telemetry against that schema and reports violations, enforcing rules such as “service names must not be broadcast IP addresses.” For teams that want earlier feedback, agentic tools can review pull requests, assess metric cardinality, and suggest instrumentation changes before code merges. Because the feedback arrives as a PR comment from a neutral automated system, engineers are more likely to act on it than on a human reviewer’s note.
Q&A
How could OpAMP help manage retention policies across collectors? Kröhling said the approach in theory would propagate new rules to the collector fleet when bad telemetry is detected, but no production-ready open source server exists yet to do this automatically. ▶ 27:24
How should agentic instrumentation tools be grounded to avoid non-deterministic output? Kröhling said off-the-shelf LLMs need a knowledge layer on top because their OpenTelemetry training data is frequently outdated, and replacing bad instrumentation with outdated semantic conventions solves nothing. ▶ 28:21
Notable Quotes
I have been taming wasteful telemetry for the past uh 5 years at least. Elena Kovalenko · ▶ 00:30
what if I need this one metric in five years time for that specific edge case now if you’ve been thinking about that case when you look at your telemetry congratulations you became a telemetry hoarder Juraci Paixão Kröhling · ▶ 05:38
7 days and 12 hours? Elena Kovalenko · ▶ 08:42
instead of emitting 1,000 logs in 10 seconds I emit one log and then I I I add a counter I’ve seen 1,000 such logs in 10 seconds. Juraci Paixão Kröhling · ▶ 18:34
there is no vendor, there is somebody else running a giant database for you. Juraci Paixão Kröhling · ▶ 25:53
Key Takeaways
- Auto instrumentation at day zero produces high-volume noise with little signal value.
- Over 700 real customer phone numbers leaked into Delivery Hero’s observability backend through a database error log.
- The OpenTelemetry Collector’s transform and log dedup processors fix PII and duplicate data without touching application code.
- OpenTelemetry Weaver enforces a telemetry schema and surfaces violations before bad data reaches the backend.
- Agentic PR review tools deliver instrumentation feedback at merge time, which engineers act on more than manual review comments.
About the Speakers
Juraci Paixão Kröhling is a software engineer at OllyGarden, a maintainer of the OpenTelemetry project, a member of the project’s governing board, and a CNCF Ambassador. He has spoken about distributed tracing and OpenTelemetry at KubeCon and related conferences.
Elena Kovalenko is a Principal Software Engineer at Delivery Hero, where she has worked on telemetry governance and observability at scale for at least five years. Her background spans backend engineering, data engineering, MLOps, and DevOps.