AI
CoreDNS Today: 400 Contributors, 14,000 Stars, and a Fresh CVE Batch
CoreDNS is a plugin-based DNS server written in Go, default in Kubernetes since 2017. The project now counts more than 400 contributors, 14,000 GitHub stars, and 29 maintainers. The most recent maintainer joined last year after making significant security contributions. Release 1.14.2 shipped two weeks before this talk. Four CVEs discovered over the previous five or six months have been addressed, and at least one more is targeted for 1.14.3, expected within a week of the talk.
Why DNS Failures Spread: The Amazon US-East-1 Lesson
A recent internet-wide outage traced back to a DNS failure in Amazon’s US-East-1 region. DNS sits beneath so many services that a single regional failure cascades broadly. CoreDNS, built in Go, benefits from memory safety that prevents whole classes of CVEs. But the language alone is not enough. Three configuration options directly address outage scenarios: prefetch keeps hot records fresh before their TTL expires, lame duck drains in-flight requests before a server shuts down, and serve-stale continues answering from cache when the upstream DNS disappears.
Serve-Stale and Multiple Upstreams: Buying Time During Outages
Serve-stale lets CoreDNS answer queries from cached records even after those records have expired, keeping services alive while an upstream is down. The Amazon outage lasted more than an hour, so serve-stale alone would not have fully covered it, but it provides time to activate a backup plan. A complementary strategy is configuring multiple upstream DNS providers. CoreDNS removes a failed upstream from its forwarding pool and routes to the remaining ones. Combined, these two options reduce the blast radius of any single provider failure significantly.
Node Local DNS Cache: From Cluster-Wide Failures to Node-Scoped Ones
Node local DNS cache runs a stripped-down CoreDNS binary as a DaemonSet on every node, eliminating the trip to the central CoreDNS for most queries. It also upgrades the node-to-central-DNS connection from UDP to TCP. UDP failures carry a 5-second timeout and leave entries in the Linux kernel connection-tracking table until they expire, filling it under load. TCP tears down cleanly. At Google, a customer with thousands of nodes saw intermittent DNS failures that took the team a long time to diagnose. Enabling node local DNS eliminated all of them.
Multisocket Scaling: Past 300,000 QPS on a Single Instance
Before CoreDNS 1.12, adding CPUs beyond two stopped improving throughput. The bottleneck was the kernel delivering all packets to a single socket. Splitting traffic across multiple sockets using the SO_REUSEPORT option fixed this. Each socket gets its own Go routines, and performance scales nearly linearly with socket count. Benchmarks with cache-only mode showed 40,000 QPS at two CPUs with one socket. With multiple sockets matched to CPU count, a single CoreDNS instance exceeded 300,000 QPS. Most clusters will never approach that ceiling, so far fewer replicas are needed than proportional autoscaling historically required.
Q&A
How should multi-tenant clusters handle one namespace flooding CoreDNS and breaking DNS for everyone? Node local DNS shrinks the failure domain from cluster to node; for persistent bad tenants, a mutating admission webhook can redirect their pods to a dedicated CoreDNS instance they can crash without affecting others. ▶ 25:43
Does running one large high-CPU node for CoreDNS introduce other limits? On some cloud providers, packet-per-second limits on a machine are hit before CPU limits, so watching network throughput caps matters as much as CPU allocation. ▶ 28:32
Notable Quotes
it’s always uh it’s always DNS as they say. John Belamaric · ▶ 05:55
we turned on node local DNS for this customer and all the failures went away. John Belamaric · ▶ 16:09
I’ve seen clusters running 80 90 nodes with, you know, with with cordns on them. That’s ridiculous, right? John Belamaric · ▶ 21:20
it’s the XKCD thing like we’re a bunch of unpaid people this is not our day job to support this but um we do it because we believe it’s important. John Belamaric · ▶ 24:47
Key Takeaways
- Serve-stale and multiple upstream providers limit the damage from a single DNS provider outage.
- Node local DNS cache upgrades node-to-DNS traffic from UDP to TCP, eliminating a class of Kubernetes DNS failures at scale.
- Multisocket scaling (SO_REUSEPORT, introduced in 1.12) pushes one CoreDNS instance past 300,000 QPS, making proportional autoscaling unnecessary for most clusters.
About the Speaker(s)
Yong Tang is Senior Director of Engineering at Ivanti and a core maintainer of CoreDNS, a role he has held since nearly the project’s beginning roughly 10 years ago. He also maintains Docker/Moby and contributes to container and machine learning projects across the open source community.
John Belamaric is a Senior Staff Software Engineer at Google, co-chair of Kubernetes SIG Architecture and WG Device Management, and co-founder of Nephio. He has worked on CoreDNS since its early days at Infoblox and continues to maintain the project as a community volunteer.