AI
The Observability Gaps at 20,000 Clusters
The DOKS team of 18 engineers runs more than 20,000 Kubernetes clusters across 13 cloud regions and promises 99.95% uptime. Last year they delivered 99.97%. At that scale, three gaps hurt: audit logs were stored in flat files that vanished on cluster restart, the kube-system namespace had no telemetry, and worker nodes required customers to SSH in before the team could diagnose anything. The goal was to catch failures before a customer filed a support ticket.
Kubeception: How DigitalOcean Structures Its Control Plane
Each customer’s Kubernetes control plane, including etcd and the API server, runs inside a separate DigitalOcean-managed cluster called the control plane cluster. Customer workloads land on data plane worker nodes in a different cluster entirely. DigitalOcean uses Cluster API to manage these customer clusters with another Kubernetes cluster, a pattern the team calls “kubeception.” Collecting telemetry across both layers, plus the worker nodes, was impossible with a single Prometheus instance per control plane cluster.
Why One Big Prometheus Instance Was Not Enough
Each control plane cluster ran one large Prometheus instance on a dedicated machine. Scaling Prometheus itself was not the real problem. The problem was coverage: Prometheus only reached the control plane layer. The kube-system namespace inside customer clusters had no metrics or logs. Worker nodes had none either. Audit logs sat in DigitalOcean Spaces object storage, unsearchable without downloading files manually. Multiple Prometheus instances at different stack layers were too expensive to justify.
The OpenTelemetry Gateway-Agent Architecture
The replacement uses the gateway-agent pattern. Lightweight agent collectors run at each layer: inside the control plane cluster scraping etcd and API server metrics, inside kube-system, and as systemd services on worker nodes collecting kubelet logs. Agents forward everything to a stateless gateway collector, which exports via OTLP to DigitalOcean’s observability backend. The OTel contrib image, with all community plugins pre-baked, made initial development fast. A transform processor replaced a dedicated Go service that had been merging telemetry fields.
Lessons Learned Migrating to OpenTelemetry
The team added the OpenTelemetry Kubernetes operator early, then found it conflicted with internal operators because both tried to manage the same deployment objects. The fix was to drop the CRD route and use plain deployment configuration instead. Security reviews, not the technical migration, extended the project timeline. The team also discovered OOM-killing collectors before they had self-monitoring in place. Adding the OTel collector’s own exported metrics as alerts from day one would have caught crashes much sooner.
Results: Hours to Minutes, No Dedicated Prometheus Machine
Mean time to detect dropped from hours to minutes. Worker node and kube-system telemetry now alerts the team directly, removing the support-ticket-to-engineering handoff loop. Searchable audit logs in OpenSearch, indexed by log type, replaced manual object storage downloads. The dedicated machine running Prometheus is gone. The gateway scales statelessly to handle export volume. Answering a customer question like “what happened three days ago” now takes seconds against a Grafana dashboard rather than a file download.
Q&A
How are the searchable logs indexed? The team uses OpenSearch internally, indexing by log type so audit logs occupy their own index and other telemetry a separate one, all ingested from OTel collectors. ▶ 25:46
Does OTel provide backpressure controls to prevent data overload? The memory limiter processor sets a burst threshold and drops data beyond it; dropped-data counts are also tracked as metrics, so alerts fire if significant telemetry is being discarded. ▶ 27:02
Did migrating to OTLP format require rewriting Grafana dashboards? The format change was mainly underscores to periods in metric names; query updates were quick and the existing Grafana dashboards needed only minor edits. ▶ 26:23
Notable Quotes
our mean time to detect issues went from hours to minutes because no longer do we have a situation Kunju Perath · ▶ 23:19
we traded some go for some YAML. Uh some people might not like that, but uh I think it it is it’s just less to maintain, less to deploy. Kunju Perath · ▶ 23:08
having observability for your observability early on is very is a very good idea to do. Kunju Perath · ▶ 20:59
we we saw hotel and we’re like all right cool there’s an open telemetry operator uh Kubernetes operator let’s just add it in and it’ll be great. Kunju Perath · ▶ 18:58
Key Takeaways
- A gateway-agent OTel pattern covered worker nodes, kube-system, and control plane in one architecture.
- Mean time to detect dropped from hours to minutes after full telemetry coverage replaced file-based audit logs.
- Adding the OTel operator too early caused operator conflicts; plain deployment config avoids that problem.
About the Speaker(s)
Yash Sharma is a Developer Advocate at DigitalOcean and a core maintainer of Meshery, a CNCF sandbox project. He mentors LFX projects, hosts weekly Meshery development calls, and spoke previously at KubeCon.
Kunju Perath is a Software Engineer on the DigitalOcean Kubernetes (DOKS) team, where he has worked for nearly four years building and operating the managed Kubernetes platform.