AI
Why Single-Cluster Kubernetes Breaks Under Real AI Inference Load
One cluster is one blast radius. A control plane incident or planned maintenance disrupts the entire inference service. GPU capacity also hits hard ceilings: when a region runs out, requests queue and P95 latency spikes immediately. Standard Kubernetes Deployments and HPA were not designed for inference-specific needs like multi-node GPU execution or graceful draining of long-lived token streams. A normal HPA scale-down interrupts in-flight requests. These two gaps, the single-cluster boundary and the vanilla Kubernetes serving gap, motivate the two-layer design.
The Two-Layer Blueprint: Karmada for Fleet, KubeRay for Serving
Karmada sits at the fleet layer and answers where workloads run. It uses declarative propagation policies for cluster affinity, spread constraints across regions, and per-cluster override policies. The Ray Service YAML stays stable; all multi-cluster topology logic lives in Karmada policies. KubeRay and Ray Serve sit inside each target cluster and answer how inference runs, covering request-driven autoscaling, health-gated upgrades, and graceful draining. Scaling and upgrades stay inside the cluster. Multi-cluster placement and resilience stay in Karmada. Neither layer bleeds into the other’s responsibility.
Karmada Fleet Patterns: Placement, Overrides, and Two-Level Failover
Override policies solve the heterogeneous fleet problem. Clusters differ in GPU node labels, storage classes, and container registries. Encoding those details inside the Ray Service YAML produces per-cluster drift. Instead, one base Ray Service definition stays untouched and Karmada applies targeted patches before propagating to each member cluster. For failures, Karmada operates two levels. Cluster-level failover triggers when a member cluster is unreachable and reschedules workloads onto healthy clusters after a configurable grace period. Workload-level failover triggers when a specific Ray Service is degraded or crash-looping, using a custom resource interpreter to map Ray Service status into health signals Karmada policies can act on.
Ray Serve: Fractional GPUs, Model Multiplexing, and Request-Driven Autoscaling
A model that needs only 30% of a GPU still occupies 100% of it under traditional serving, leaving 70% idle. Ray Serve supports fractional GPU allocation, so that model requests exactly 30% and the remainder is available to other workloads. GPU utilization can jump from 30% to near 100% with one configuration change and no specialized hardware. For model variants sharing the same architecture but carrying different weights, Ray Serve supports model multiplexing: one replica hosts multiple variants in GPU memory, loading and evicting on demand. With LoRA adapters, only the adapter weights, typically less than 1% of the base model, swap in and out, keeping swap latency and memory footprint small.
KubeRay Autoscaling: Three Layers From Replicas to Kubernetes Nodes
Inference traffic can go from near zero to thousands of requests in seconds. KubeRay autoscaling operates across three layers. Layer one is the Ray Serve autoscaler, which scales replicas based on queue depth and ongoing requests, not CPU utilization, so it reacts faster to bursty inference traffic than default HPA. Layer two is the Ray autoscaler: when the cluster lacks capacity for new replicas, Ray adds worker pods targeting the right worker group, GPU workers for LLM deployments, CPU workers for preprocessing. Layer three is the Kubernetes cluster autoscaler, which provisions new nodes when Ray worker pods cannot be scheduled. Custom autoscaling policies are also supported when the built-in behavior is insufficient.
KubeRay Ecosystem Features: History Server and Scheduler Integration
KubeRay integrates with Kueue, Volcano, Apache YuniKorn, and Nvidia’s scheduler plugin for gang scheduling and priority queues. Ray metrics export to Prometheus with pre-built Grafana dashboards covering cluster health, autoscaling, and request latencies. The new history server, co-developed by Anyscale, Google GKE, and Alibaba ACK, fixes a common debugging gap: when an ephemeral cluster tears down after a job failure, the Ray dashboard disappears. The history server continuously writes cluster metadata and logs to S3 or GCS. After deletion, it restores the full dashboard from that data, allowing post-mortem debugging with no live cluster cost.
Notable Quotes
A model that only needs 30% of a GPU still occupies 100% of it leaving 70% idle. Han-Ju Chen · ▶ 12:35
So you can register hundreds of model variants but only keep a handful warm at any time. Han-Ju Chen · ▶ 14:11
The history server fix this. While a cluster is alive, it automatically collects cluster meta data and logs into persistent storage like S3 or GCS. Han-Ju Chen · ▶ 28:00
A single cluster is a hard boundary and val primitives do not provide the serving life cycle inference needs. Han-Ju Chen · ▶ 04:48
Key Takeaways
- Karmada propagation and override policies keep one Ray Service YAML across heterogeneous clusters, eliminating per-cluster drift.
- Ray Serve fractional GPU allocation can push utilization from 30% to near 100% with one config change.
- Three-layer autoscaling, replicas, Ray worker pods, and Kubernetes nodes, reacts to queue depth rather than CPU metrics.
About the Speaker(s)
Wei-Cheng Lai is a software engineer on Bloomberg’s AI Platform Engineering team, building Kubernetes-based infrastructure for AI applications. He contributes to both Karmada and KubeRay, with a focus on multi-cluster orchestration.
Han-Ju Chen is a software engineer at Anyscale, focused on KubeRay. He brings prior experience from two years contributing to Flyte and works across cloud-native MLOps infrastructure.