AI
The Problem: 5,000 Nodes Sending 10 MB Objects
Kubescape stores deep eBPF runtime data in container profiles. A single profile runs to tens of megabytes. At 5,000 nodes, each sending 10 MB objects for every running container on a periodic cycle, the result is a thundering herd. Objects too large for etcd forced a custom aggregated API server. That server became the bottleneck. Thousands of node agents hammering the same storage endpoint produced lock contention that made standard Kubernetes storage patterns stop working.
Attempt One: Full Object Writes Every 10 Minutes
The first design kept the entire CRD in memory on each node agent and flushed it every 10 minutes using the create method. With 5,000 agents running as a DaemonSet, all nodes wrote simultaneously. Conflicts forced each agent to fetch the current object, merge local data, and repost, generating repeated round-trips. Lock contention made this pattern unworkable at scale. The back-and-forth traffic alone was enough to degrade the API server under normal cluster load.
Attempt Two: JSON Patch and the 30-Second Wall
The second attempt switched to JSON patches. Each node agent computed a diff every 10 minutes and sent only the delta. The aggregated API server applied it server-side. The problem is synchronous execution. The server had 30 seconds to read the large object from disk, unmarshal it, apply the patch, and write it back. Mutexes to prevent concurrent updates on the same object made that deadline impossible to meet at scale. The approach failed for the same underlying reason as the first.
Attempt Three: Time-Series Writes With Async Aggregation
The current architecture moves processing out of the write path. Each node agent now sends a unique CRD per period rather than updating a shared object. The API server treats these as a time series, writes each entry to disk immediately, and returns an acknowledgement. Aggregation happens later, during low CPU utilization, as a separate asynchronous step. Decoupling the write path from the merge logic removed the lock contention entirely and allowed Kubescape to scale across thousands of nodes.
What Comes Next: Coordinated Learning
Bertschy outlined a planned fourth phase. When thousands of nodes observe the same containers, they generate redundant data. The storage layer could signal that only five learners are needed for a given workload, letting the rest wait for results rather than submitting duplicate profiles. This would cut write volume further without changing the node agent interface. The feature is not shipped yet, but the decoupled architecture built in phase three makes it straightforward to add.
Notable Quotes
What you end up with is a thundering herd Matthias Bertschy · ▶ 00:27
when you have uh different replicas or even more like demon sets, you have 5,000 node agents sending the same data at the same time and you have a lock contention Matthias Bertschy · ▶ 02:27
you only have 30 seconds to read the big object from the disk unmarshall it apply the patch and save it back Matthias Bertschy · ▶ 03:19
Key Takeaways
- At 5,000 nodes, synchronous writes to a shared CRD produce fatal lock contention.
- Treating periodic profiles as time-series entries decouples writes from aggregation and removes contention.
- A future phase will let storage cap active learners, eliminating redundant writes from identical workloads.
About the Speaker(s)
Matthias Bertschy is a Senior Kubernetes Developer at ARMO and a principal maintainer of Kubescape. He joined a Swiss security solution provider in 2011, earned a GPEN ethical hacker certification in 2012, and a CISSP in 2015. He is a member of SIG Node and a reviewer for the kubelet and test-infra. Before ARMO, he spent five years helping Swiss banks with digital transformation as a Senior DevOps Engineer and Platform Architect.