AI
What Profiles Are and Why They Matter
Profiling has three main uses: incident diagnosis, cost reduction, and performance tuning. During an incident, a CPU profile can show which code path turned hot, for example, a sudden increase in JSON parsing that points to larger payloads. At hyperscaler spend levels, profiling identifies where CPU and memory go. For tail-latency work, the OpenTelemetry format attaches trace and span IDs to each CPU sample, letting engineers ask whether P99 requests follow a different code path than P50 requests.
How the OpenTelemetry Profile Signal Is Structured
The profile signal shares the same resource and scope concepts as logs, metrics, and traces, but adds structures specific to profiling. A profile dictionary stores every string once in a string table and uses references everywhere else. That single change cuts wire size by up to 40%, meaningful for continuous profiling where data flows constantly. A new first-class link message carries span ID and trace ID, directly correlating each profile to other signals. The format originated from pprof but adds deduplication and timestamp support that pprof handles poorly.
The eBPF Collector: No Instrumentation Required
The collector runs as a DaemonSet, one instance per node, not one per pod. It installs eBPF programs and maps in the Linux kernel to unwind stack traces, then symbolizes them in user space before sending to any OTLP-compatible backend. No application restart or code change is needed. CPU overhead sits around 1%. Custom unwinders cover Java, V8, PHP, Node.js, and Perl in addition to native languages. Version 0148 of the collector is the first build that supports the profiling alpha, and enabling it requires a feature gate flag that will disappear at stable.
Live Demo: Verifying the Profiler Against Known Ground Truth
Geisendörfer used a folded-stack text file as ground truth, prompted an LLM to generate a Python program that burns CPU in exactly those proportions, then ran it on a remote Linux machine tunneled over SSH. The eBPF profiler picked up the Python process and rendered the correct flame graph, including the call from Python’s time function through libc into VDSO. He then generated an equivalent Node.js program on the spot. The profiler captured both simultaneously, showing mixed native and JavaScript stacks in a single view, something single-language profilers cannot do.
Real Incident: AI Refactoring Bug Caught by Flame Graph Diff
A pre-recorded e-commerce scenario showed a v2 deploy that dropped errors to 0% but sent latency and CPU upward, with memory climbing without bound. Comparing before-and-after flame graphs surfaced a capture_error_context call inside process_orders that was absent before the deploy. The root cause: an AI refactoring tool changed a break to continue, creating an infinite loop whenever an item was out of stock. That 1% background error rate became a spin loop. This was not a contrived example; Geisendörfer noted a real company encountered the same bug and used profiling to find it.
Q&A
Is there an SDK-based path to emit profiling signals beyond eBPF? A pprof receiver already converts Go pprof data to OTLP profiles, a Go flight recorder donation proposal is in progress, and async profiler can output OTLP profiles from Java. ▶ 29:13
What does the OTel profile format offer over pprof? Stack traces are first-class objects referenced by a single ID, so timestamps do not require repeating the full stack; pprof has no equivalent, and pprof converts to OTLP and back without data loss. ▶ 31:20
Does the trace-span link field work today? Any emitter that populates the link message will have working correlation; the team is also building eBPF-level context sharing with Beyla so the link field is populated automatically. ▶ 32:42
Notable Quotes
dduplication was a high priority for the profiling sick uh because you want to be as most efficient as possible Florian Lehner · ▶ 09:40
you just deploy it and it just works. So that’s a key advantage and key feature. Florian Lehner · ▶ 12:39
this was actually a real production instance from a from a company using LLM to produce LLM software. This is where we are today. Felix Geisendörfer · ▶ 27:26
30 OTLP pull requests were merged and many more commits to the collector EBPF profiler and other repositories were done to make this happen. Felix Geisendörfer · ▶ 06:45
Key Takeaways
- The profile dictionary cuts OTel profiling wire size by up to 40% over repeated-string formats.
- The eBPF collector profiles any language at roughly 1% CPU overhead with no application changes.
- Flame graph diffs between deploy windows can surface infinite loops before memory exhaustion crashes a service.
About the Speaker(s)
Felix Geisendörfer is a Senior Staff Software Engineer at Datadog working on continuous profiling for Go. Before Datadog he built manufacturing systems for Apple and managed large PostgreSQL clusters.
Florian Lehner is a Software Engineer at Elastic focused on performance and security. He works on eBPF-based observability and continuous system profiling.