AI
Two Kinds of AI Workloads on Kubernetes
Training and inference jobs pack model weights and data into images, rely on fast inter-node communication libraries, and need GPUs or high-performance networking that takes time to allocate. They run as groups of pods that start and stop together. Agent sandboxes have different demands: sub-second creation, file system access for token injection and code execution, and lifecycle operations (hibernate, resume) that previously lived in virtual machines.
In-Place Update Cuts Inference Downtime
Kubernetes normally updates a pod by deleting and recreating it, which reassigns IPs, moves the pod to a new node, and forces fresh volume mounts. In-place update skips the pod delete. OpenKruise deletes only the target container, so Kubernetes restarts just that container on the same node. The node reuses its existing IP, volume mounts, and sidecar containers. Image pulls also get faster: Docker layers from the previous image stay cached, so only changed layers download.
Image Prewarming for Large AI Images
AI images now bundle model weights alongside application binaries, making them large enough to slow pod scheduling. OpenKruise’s ImagePullJob custom resource accepts a list of images, a parallelism limit, and node or pod selectors. The OpenKruise daemon on each matching node pulls images in advance, so pods land on nodes where images already exist. For scheduled training jobs with known start times, OpenKruise’s AdvancedCronJob wraps an ImagePullJob template and uses cron syntax to trigger pre-pulls before the job runs.
Pod Group Disruption Policy for Multi-Node Inference
Large inference services span multiple nodes, with pods forming groups where each group must run together or not at all. If one pod in a group fails, the whole group becomes unavailable. A voluntary eviction of a pod in a different group then breaks the minimum capacity for the entire service. OpenKruise adds a pod group policy: a label marks which pods belong to the same group. If group zero is down, the pod disruption budget blocks voluntary evictions from groups one and two, keeping remaining inference capacity intact.
OpenKruise Agent: Sandbox Allocation in 20 Milliseconds
OpenKruise Agent is a new sub-project targeting AI agent sandbox infrastructure. A warm pool of pre-allocated sandboxes lets agents acquire a new sandbox in under 20 milliseconds, faster than lightweight microVMs. The sandbox controller handles hibernation: when a sandbox sits idle waiting for user input or a task signal, the controller saves the root filesystem and memory to shared storage, cutting cost during idle periods. Checkpoint goes further, snapshotting a running sandbox and cloning it into multiple replicas for pre-upgrade backups or fast application startup.
Q&A
What are the storage and IO costs of image prewarming on local nodes? Pre-warming occupies local disk space and increases IO load on the node’s disk. ▶ 19:15
Notable Quotes
almost no image downloading uh required Zhen Zhang · ▶ 6:54
we just in place restart these Zhen Zhang · ▶ 12:32
we can get a samples in almost 20 Zhen Zhang · ▶ 15:48
Key Takeaways
- In-place update avoids pod deletion, reusing the node’s IP, volume mounts, and sidecar containers during inference upgrades.
- ImagePullJob pre-pulls model images to matching nodes before pods schedule, eliminating download time at runtime.
- Pod group policy blocks voluntary evictions when another pod group is already unavailable, protecting inference capacity.
- OpenKruise Agent provisions sandboxes in under 20 milliseconds from a warm pool, faster than lightweight microVMs.
- Checkpoint snapshots a running sandbox and clones it into multiple replicas for pre-upgrade backups or fast startup.
About the Speakers
Zhen Zhang is a Staff Engineer at Alibaba Group focused on cluster management and application orchestration. He drives cloud native innovation inside Alibaba and serves as one of the main maintainers of the OpenKruise project.
Vec Sun (Sunweixiang) is a CloudNative Developer at Xiaohongshu (RedNote). He previously worked on the Alibaba Cloud container team and contributes to OpenKruise, Karmada, and related communities, with deep focus on container application orchestration and multi-cluster management.