AI
A Security Patch Shipped Friday: What etcd-auth Users Need to Know
A security release went out the Friday before this talk. The bug is in etcd auth, an optional authentication and authorization layer used when etcd runs outside the Kubernetes control plane. The Kubernetes control plane does not use etcd auth to control access to etcd itself, so most operators are unaffected. Anyone running etcd auth should upgrade immediately. Everyone else can wait for the next scheduled maintenance window.
etcd 3.6: Official Downgrade and the End of the v2 Store
etcd 3.6 makes downgrades officially supported for the first time. The process validates cluster health, then enables a rolling downgrade one member at a time. Version attributes in the raft protocol and write-ahead log enforce a one-minor-version limit, so 3.6 can roll back to 3.5 but no further. etcd 3.6 also completes the removal of the v2 store. The v3 store is now the sole source of truth, and snapshot frequency drops from 100,000 log writes to 10,000, making processing faster and less data-intensive.
etcd 3.7 Roadmap: Paginated Range Queries and Protobuf Cleanup
The headline feature for 3.7 is etcd range pagination, tracked as KEP 5967. Today, querying a million keys locks the client and can crash it. The fix chunks results and streams them to the client, the same pattern used across the database industry. Beyond that, 3.7 will remove archived Protobuf libraries, drop experimental gRPC features, and ship a new client-side load balancer to replace the experimental upstream one etcd currently depends on. Two performance proposals, raft async writes and a configurable bbolt rebalance threshold, remain blocked on finding a use case where they actually help.
etcd Operator 0.2: Live Demo of Certificate Management
The etcd operator released version 0.2.0 a few weeks before this talk. It manages etcd clusters for application workloads inside Kubernetes, not the Kubernetes control plane itself. The live demo at this session spun up a kind cluster, deployed the 0.2 operator image from the official Kubernetes registry, and applied a cluster object specifying the etcd version, cluster size, and TLS provider. With auto TLS, client, peer, and server secrets were mounted to each member as it came up. A second demo showed the same flow with cert-manager supplying a namespaced or cluster issuer.
Support Windows, Official Images, and How to Contribute
etcd 3.4 stops receiving patches in two months. etcd 3.5 is supported until at least next May, possibly longer if 3.7 slips. Starting with Kubernetes 1.35, new Kubernetes installations pull the official etcd images from the Kubernetes registry. The project is actively recruiting: open roles include a new lead for the raft sub-project, performance engineers for async write and bbolt work, and contributors for the Protobuf overhaul. The docs team is reorganizing documentation into a role-based structure matching Kubernetes and may drop versioned docs to reduce backport burden.
Q&A
Can etcd limit storage per namespace to prevent one tenant from filling the database? Kubernetes 1.34 adds a per-resource size metric, but per-namespace limits are unsupported in etcd today and the maintainers called multi-tenancy in etcd an open, hard problem. ▶ 28:53
Notable Quotes
if you are using CD off you should probably upgrade after we get done with this talk. Josh Berkus · ▶ 02:28
we don’t have a good multi-tenency story it’s something that we are like I know considering but it’s very uh Uh I know I would say pretty hard because everyone uses Kubernetes and doesn’t use like multi-tenency. Merrick · ▶ 32:07
unlike some other databases, CD feels like if a knob isn’t going to do anything, we won’t expose it to the users. Josh Berkus · ▶ 14:29
for a distributed consistent database there is literally no limit to the amount of robustness testing you could potentially do. Josh Berkus · ▶ 26:45
Key Takeaways
- etcd 3.4 loses patch support in two months; plan your upgrade now.
- Downgrade from 3.6 to 3.5 is now officially supported, one member at a time.
- etcd 3.7 needs performance engineers and a new raft sub-project lead to ship its roadmap features.
About the Speaker(s)
Josh Berkus is the Kubernetes Community Manager at Red Hat’s Open Source Program Office. He has contributed to Linux, OpenOffice, PostgreSQL, and many other projects over more than 20 years, and currently works on Kubernetes, etcd, Elekto, Podman, and uBlue.
Arka Saha is a Software Engineer at Broadcom who leads Kubernetes releases and maintenance for VMware Cloud Foundation. He manages Broadcom’s Prow infrastructure and supports long-term maintenance of Kubernetes, etcd, containers, and Golang components. He joined this talk as a recently promoted etcd operator reviewer, having come through the etcd mentorship cohort.