AI
The Span Explosion in a Single Request
One HTTP request through an Express JavaScript service to PostgreSQL generates 8 spans under auto instrumentation. Costa and Oliveira argue you can act on two: the incoming request and the database query. The other six, including request handler spans that duplicate root trace information, pool connect, PG connect, and DNS lookup spans, are noise. At 10 services this is manageable. At 100 or 200 services running the same configuration, the volume compounds proportionally.
Resource Cost Measured Across Two Languages
The team ran K6 at 10,000 requests per second for 15 minutes against both apps. In Express, auto instrumentation used roughly 40% more CPU and 34% more memory than manual. Java showed 37% more CPU but only 12% more memory. Storage told the clearest story: auto-instrumented Express generated 44 GB more trace data than the manual version in that 15-minute window, even with identical request counts. Java added 42 GB. Both services traced the same number of requests; auto instrumentation added more spans per trace.
Sampling Does Not Solve the Root Problem
Oliveira framed the real sampling decision: 100 auto-instrumented services at 1% sample rate produce similar storage costs to 100 manually instrumented services at 80%. The auto side carries much higher CPU and memory overhead on each application instance. At 1% sampling you discard 99 of every 100 traces. At 80% with manual instrumentation you keep most of what you produce. The storage bill may look the same on paper, but resource consumption during trace generation does not.
Auto Instrumentation Leaks PII by Default
Auto instrumentation captures request URLs and bodies without filtering. The demo showed a songs endpoint where query parameters included artist names that were not sanitized. A URL containing a social security number ships that number directly to the APM backend. The OTel Collector can redact it with a regex transform processor, but that means producing the data first and paying CPU to scrub it afterward. Costa’s point: if the data serves no business purpose, do not generate it.
A Framework for Intentional Instrumentation
Costa laid out a four-step order: define observability goals before any instrumentation work, instrument with purpose against those goals, validate the data in your backend after deploying, then consider sampling based on traffic volume. Precise data reduces mean time to detect and mean time to recovery. That reduces on-call burden. Oliveira closed with a metric most cost conversations skip: intentional observability correlates directly with team mental health and customer satisfaction.
Q&A
How can a platform team minimize developer instrumentation involvement while keeping useful granularity? Start with full auto instrumentation, inspect what the backend captures, then suppress specific instrumentations in the SDK. For JavaScript, configure suppressions at Node SDK setup. For Java, use environment variables. You keep automatic coverage but stop shipping spans that don’t serve you. ▶ 20:41
Manual instrumentation often breaks semantic conventions. How do you enforce them? Costa recommended OTel Weaver, an OpenTelemetry sub-project that plugs into CI/CD and validates telemetry against semantic conventions before it reaches production. ▶ 22:19
What about threshold-based sampling, where a TCP connect span is kept only if it runs unusually long? Use tail sampling in the OTel Collector based on latency. Slow traces pass through a pipeline that keeps them; fast traces pass through a pipeline that drops the noisy spans. ▶ 23:53
Can you combine manual instrumentation for business logic with selective auto instrumentation for third-party libraries like DNS? Yes. OTel instrumentation libraries let you selectively auto-instrument specific libraries, giving you control over exactly what gets traced without enabling everything. ▶ 26:15
Notable Quotes
we have eight spans Juliano Costa · ▶ 3:53
I don’t believe so. Juliano Costa · ▶ 14:34
saving you more than just money. Juliano Costa · ▶ 19:58
Key Takeaways
- A single Express HTTP-to-DB request generates 8 spans; auto instrumentation defaults to all of them.
- Auto instrumentation adds 40% CPU and 34% memory overhead versus manual in Express JavaScript.
- Identical storage costs at 1% auto sample rate versus 80% manual rate mask a large resource gap.
- Auto instrumentation ships raw URLs and request bodies to your APM backend by default, including PII.
- OTel Weaver enforces semantic conventions in CI/CD before telemetry reaches production.
About the Speaker(s)
Juliano Costa is a Developer Advocate at Datadog focused on OpenTelemetry. He is a CNCF Ambassador, an active contributor to the OpenTelemetry project, and a maintainer of several OTel components.
Yuri Sa (Yuri Oliveira) is co-founder of OllyGarden and a Senior Software Engineer with 15 years of experience in critical infrastructure as a SysAdmin, SRE, and DevOps Engineer, focused on helping companies reach the next level of observability automation.