AI
Why Coupled Prefill and Decode Hurt GPU Utilization
LLM inference runs in two distinct stages. Prefill processes the entire prompt at once: highly parallel and compute-bound. Decode generates tokens one by one: memory-bandwidth-bound. Running both on the same GPU instance causes interference. A long-prompt request A fights request B for compute resources on the same silicon, producing latency spikes and leaving expensive hardware underutilized. User experience is measured by two metrics, TTFT (time to first token) and TPOT (time per output token), and a coupled architecture forces a direct trade-off between them.
The XPYD Model: Splitting the Cluster by Stage
Prefill-decode disaggregation divides the cluster into two typed pools. Pool P handles prefill; pool D handles decode. The XPYD label names the ratio: X prefill instances to Y decode instances. A RAG pipeline that processes massive documents but returns short answers scales X up. A coding assistant generating hundreds of lines of code from a short prompt scales Y up. Each pool pays only for the hardware profile its stage needs. The catch: once prefill finishes, it must transfer the KV cache over the network to the decode node, and for large contexts that cache can reach gigabytes.
Vanilla Kubernetes Cannot Manage XPYD Topology
Standard Kubernetes deployments have no concept of prefill or decode pools. Each requires a separate deployment and a separate node group definition. The default HPA cannot scale on KV cache pressure. Worst of all, the default scheduler is topology-blind: it might place the prefill pool on rack one and the decode pool on rack ten. The KV cache then travels across multiple switches, and network latency erases every TTFT gain from disaggregation. That operational gap is why the Volcano community built Casina.
Casina: Purpose-Built Orchestration for Disaggregated Inference
Casina is an open-source sub-project of Volcano with two decoupled components. The Casina router handles data-plane traffic routing. The Casina control manager is an operator that manages distributed PD-disaggregated workloads. Three lightweight CRDs define the system: ModelServer selects inference engine instances, ModelRouter defines traffic rules, and ModelServing groups everything into an XPYD topology. A developer sees one model endpoint and does not need to know whether it runs across ten machines or two distinct pools. The PD group attribute in ModelServer tells the router which pods are prefill and which are decode, with no sidecars required.
Two-Level Scheduling and Cache-Aware Routing
Casina integrates with Volcano for two-level gang scheduling. At the serving-group level, at least one prefill and one decode instance must be scheduled together. At the row level, all pods within a row are co-located, keeping KV cache transfers within the same hypernode. The Casina router continuously monitors backend inference engines such as vLLM and SGLang, tracking pending queues, KV cache usage, and GPU utilization. Sending requests to pods that already hold the relevant context in memory, combined with a least-request strategy, cuts time-to-first-token by nearly three quarters compared to random assignment.
Roadmap: Version 0.3 and Heterogeneous Accelerator Support
Version 0.3 made Casina production-ready with four inference framework integrations and end-to-end router observability. Upcoming work includes advanced scaling plugins for semantic routing, heterogeneous accelerator support so operators can mix GPUs and NPUs across prefill and decode pools, and zero-downtime upgrades for disaggregated topologies. The Kubernetes Gateway API and the Gateway API inference extension are already supported, so teams can manage disaggregated LLM workloads using the same traffic management patterns they use for microservices.
Q&A
How does Casina handle expert parallelism in mixture-of-experts models like DeepSeek? Expert load balancing across individual GPUs is the responsibility of the inference engine (such as vLLM), not Casina; Casina focuses on high-level routing between prefill and decode pools, not internal expert routing. ▶ 31:15
Notable Quotes
the only way to break this trade-off is to ripe this two stages apart which leads to uh prefill decode disagregation. Zhonghu Xu · ▶ 06:05
the default Kubernet Kubernetes scheduleuler is topology blind. It might it might sped your prefer port on rack one and then uh your decode port on rack 10 killing your K catch transfer speed. uh this massive operational gap is exactly why we built casina. Zhonghu Xu · ▶ 10:14
And the time it takes to get the first token drops by nearly three quarters. Zhonghu Xu · ▶ 24:13
for the EP load balance. Actually, it is the responsibility of the inference engine like VM. Zhonghu Xu · ▶ 31:15
Key Takeaways
- Coupled prefill-decode on one GPU forces a direct trade-off between TTFT and TPOT.
- The XPYD model enables independent scaling of prefill and decode pools per workload type.
- Cache-aware routing in Casina cuts time-to-first-token by nearly three quarters versus random assignment.
- Two-level gang scheduling co-locates prefill and decode pods to minimize KV cache transfer latency.
- Casina v0.3 ships production-ready with four inference framework integrations and Gateway API support.
About the Speaker(s)
Zhonghu Xu is a Software Architect at Huawei Technologies. He is an Istio Steering Committee member and one of Istio’s top three contributors, having joined as a core maintainer from the project’s early days. He serves as CNCF TAG-Infra Tech Lead and maintains several CNCF projects including Istio and Kmesh.