CRI List Streaming: Breaking the 16 MB gRPC Wall

▶ Watch (2:38)

On nodes running 10,000 or more containers, the kubelet’s list operations fail completely. The CRI list RPCs serialize every result into a single protobuf message, and the kubelet enforces a 16 MB hard limit. That limit has already been raised twice, from 4 MB to 8 MB to 16 MB, and raising it again just defers the problem. The fix, entering alpha in Kubernetes 1.36, introduces server-side streaming RPCs. The runtime sends results in chunks. The kubelet falls back to the unary RPC if the runtime returns unimplemented, so upgrade order does not matter.

CRI Stats: Ending the cAdvisor Duplication

▶ Watch (5:37)

Today the kubelet and the container runtime both read the same kernel interfaces and maintain separate in-memory state for CPU, memory, filesystem, and network usage. CRI stats fixes this by having the kubelet ask the runtime directly via CRI RPCs. CRI-O implemented the required metrics for the beta graduation target in Kubernetes 1.36, but benchmarks showed no improvement because the cAdvisor housekeeping code path still runs even when CRI stats is enabled. Removing that path is planned now. The feature stays in alpha until that kubelet-side work lands.

Memory QoS: Closing the Gap Between Scheduler and Kernel

▶ Watch (13:08)

The kubelet sets OOM scores to steer the kernel toward killing best-effort pods first, but that hint does not reach the kernel’s memory reclaim path. A guaranteed pod requesting 2 GB can still have its pages reclaimed, spiking latency regardless of QoS class. Memory QoS tells the kernel to protect that allocation. The feature was introduced in Kubernetes 1.22, disabled in 1.27 due to a livelock bug in kernels before 5.9, and retargeted at 1.36 alpha. The revised design uses memory.min only for guaranteed pods and memory.low for burstable, skipping protection entirely for best-effort.

Additional Artifact Stores: Placing OCI Artifacts on Separate Storage

▶ Watch (7:56)

CRI-O 1.35 shipped a new configuration option, additional artifact stores, under the CRI runtime section. Previously all OCI artifacts landed in a single hardcoded path under the graph directory. The new option defines additional read-only directories checked in order before the default. If an artifact is found, CRI-O uses it without copying. The primary targets are machine learning workloads where models run 20 to 50 GB and belong on dedicated NVMe or NFS volumes, air-gapped environments where nodes boot with pre-baked artifacts, and shared read-only mounts populated once and consumed by every node in the cluster.

Peer-to-Peer Image Distribution with Spiegel

▶ Watch (20:41)

A 100-node GPU cluster pulling a 20 GB LLM model generates roughly 2 TB of egress per rollout. Spiegel is a stateless daemonset that uses a distributed hash table and a P2P protocol to serve image layers from local network peers instead of a remote registry. When CRI-O pulls an image, Spiegel intercepts the request, checks whether a peer already has the relevant blobs, and fetches locally. The demo showed worker-one pulling an image already present on worker-zero, with journal logs confirming the pull came from localhost port 500. The reported bandwidth saving is 90%. CRI-O required a new block cache layer because it stores decompressed layers and Spiegel needs the compressed originals.

Notable Quotes

The limit has already been raised twice from 4 megabytes to 8 megabytes to 60 megabytes. But raising it again would just be kicking the can down the road. Ayato Tokubi · ▶ 03:46

So you end up with two components on the node both reading the same kind of interfaces both maintaining their own state and occasionally disagreeing on the numbers. That’s not a great architecture. Ayato Tokubi · ▶ 06:09

it is use local 500 to pull that image. Sohan Kunkerkar · ▶ 29:28

with P2P without P2P with P2P one image pull and 99 peer-to-peer Sohan Kunkerkar · ▶ 29:44

Key Takeaways

  • CRI list streaming enters Kubernetes 1.36 alpha, fixing failures on nodes with 10,000-plus containers.
  • CRI stats stays alpha because the kubelet still runs cAdvisor alongside it, paying both costs simultaneously.
  • Spiegel’s P2P distribution cuts egress by 90%, with CRI-O integration requiring a new compressed block cache layer.

About the Speaker(s)

Sohan Kunkerkar is a Senior Software Engineer at Red Hat with expertise in distributed systems, backend engineering, and containers. He is a CRI-O maintainer and active contributor to Kubernetes Sig-Node, and also contributes to Kueue, the queuing system for batch workloads.

Ayato Tokubi is a software engineer at Red Hat specializing in container technologies. He is a CRI-O maintainer and has contributed to Kubernetes and various other open-source projects.