AI
The DNS Blind Spot in Existing Instrumentation
Standard SDK and agent instrumentation only starts timing after DNS resolves. A service can fire thousands of DNS queries, route through multiple search domains, and still succeed, so no error fires. The latency just appears as slow startup or a sluggish outgoing call. Endre Sara described customers generating millions of NX domain responses per day from Kafka bootstrap alone, with no visibility into which client was responsible or why the extra round-trips were happening.
How OB Captures DNS at the Kernel Level
OB hooks into the Linux kernel and taps both UDP and TCP on ports 53 and 5353. It matches DNS request and response packets using the DNS ID field, then extracts the process that made each request. That raw event travels through a ring buffer to the OB user-space component, which enriches it with Kubernetes API metadata: pod name, service name, namespace, and cloud vendor labels. The final telemetry ships as OpenTelemetry metrics or traces, with no changes needed in the application.
Two Telemetry Signals: Metrics and Trace Spans
The primary signal is a histogram metric, dns.lookup.duration, annotated with the queried name, service name, namespace, and Kubernetes metadata. Trace spans are optional and off by default. When enabled, a DNS span nests as a child of the parent HTTP or gRPC span, before the outgoing client call. Because traces do not suffer cardinality explosion, span records also include resolved IP addresses and how many DNS calls were needed to get a response. One example trace showed two DNS calls per request, one A record and one AAAA, because the client library skipped IP caching.
Coexistence with Existing OTel Instrumentation
OB watches for OTLP trace and metric exports from each service. If a service already exports its own telemetry, OB skips duplicating those signals for that service. DNS is a signal no existing SDK emits, so OB adds it on top regardless. A Java service instrumented with the OTel Java agent, a Node.js service using auto-instrumentation, and an uninstrumented Go service on the same node all receive DNS metrics automatically. The Go service also gets HTTP metrics and traces from OB with no SDK work.
Security Model for a Privileged Daemon
OB requires elevated Linux capabilities to read network data, but it runs entirely outside application process space. The OB container image ships with no shell. It exports no ports. A CVE in an SDK bundled inside an application affects that application’s process; a CVE in OB affects only the monitoring daemon. Nikola Grcevski noted that if OB misbehaves, say by stressing a garbage collector, it cannot touch the application’s P99 or P999 latency figures because it runs separately. OB can be locked to a dedicated namespace accessible only to a small operator group.
Q&A
Does OB require privileged: true in the container spec? Not necessarily. OB works with specific Linux capabilities rather than full privilege, and the project publishes a recipe listing exactly which permissions are needed. ▶ 25:43
Can OB correlate CoreDNS OTel spans with client-side DNS spans in the same trace? Not fully yet. OB can find the trace ID from observed traffic and emit a span, but correct nesting inside custom spans is not guaranteed. The team acknowledged this as a direction worth exploring. ▶ 26:41
Can the port filter be extended beyond 53 and 5353? Port 53 and 5353 are currently hardcoded, but a config option is feasible. The speakers asked attendees to open a GitHub issue. ▶ 27:34
Is OB production-ready given the “in development” label in the repository? A release candidate is committed for 2025. Endre Sara added that Causely already runs OB across thousands of nodes across many customers. ▶ 28:45
Notable Quotes
it’s never DNS. It’s always DNS. Endre Sara · ▶ 02:14
the problem is actually not in the server, the problem is in the client. So, how do you know which client is causing the DNS overload so that you can actually fix the problem rather than trying to just infinitely scale your DNS infrastructure? Endre Sara · ▶ 12:55
I never realized that I used to make 100 DNS requests, and now I’m making 1,000 DNS requests a second from this particular client. Endre Sara · ▶ 32:15
Like the OB image itself, you cannot even log in. There’s no shell in there. Nikola Grcevski · ▶ 23:03
Key Takeaways
- DNS lookup duration histograms, attributed per pod, require zero code changes to collect.
- OB detects existing OTel exporters and adds DNS metrics without duplicating other signals.
- Out-of-process eBPF instrumentation cannot degrade application P99 latency or inherit SDK CVEs.
About the Speaker(s)
Nikola Grcevski is a Principal Software Engineer at Grafana Labs and a maintainer of the OpenTelemetry eBPF instrumentation project. He has spent more than 20 years in compilers, managed runtimes, and performance optimization, and now focuses on low-level application instrumentation with eBPF.
Endre Sara is Co-Founder of Causely, where he is building a causal reasoning layer for IT operations. Previously he was VP of Advanced Engineering at Turbonomic and a VP at Goldman Sachs. He holds an M.E. in Electrical Engineering and has collaborated with the OB project to drive DNS observability from the client side.