AI
Familiar Kubernetes Patterns Applied to GenAI Workloads
Six patterns visible in a standard web app on Kubernetes, including the controller, init container, stateful service, batch job, and daemon service patterns, map directly onto GenAI workloads. The controller pattern shifts to KServe, a CNCF incubating project. You declare an InferenceService, specify the model, the source, and the runtime (vLLM, TGIS, etc.), and KServe creates the deployments, services, scaling rules, and routing policies. A Postgres replica becomes a vector database. A reporting batch job becomes a document chunking and embedding job that populates that vector database.
Where Existing Patterns Change at GenAI Scale
Some patterns apply almost unchanged; others shift meaningfully because the parameters change. Health probes that previously ran in seconds now need multiple minutes, and they must confirm model weights are loaded and the key-value cache is allocated. Rolling deployments become harder because GPU nodes are scarce, so a fixed deployment with brief downtime is often the better call. Batch jobs for fine-tuning benefit from Kubernetes gang scheduling, which holds the job until all tasks can run simultaneously. Startup times of up to 10 minutes reshape how you plan declarative deployments.
Model Data Initialization: From Init Containers to Image Volumes
Downloading 50 GB of model weights on every pod start does not scale. Three approaches exist. An init container can pull weights from S3 or copy them from a baked OCI image into a shared emptyDir volume, but you pay the copy penalty every time or on every pull. A pre-populated persistent volume removes repeated downloads but puts model data on a remote filesystem, which adds latency. The best current option is image volume types, enabled by default in Kubernetes 1.35. You declare a volume with an image field referencing an OCI image and mount it directly, no copy required.
The ModelCar Pattern: A Workaround Before 1.35
Before image volumes landed, operators used a pattern called ModelCar. A sidecar container holds the model image and runs sleep infinity. Enabling shared process namespace lets the main container reach the sidecar’s root filesystem via /proc, so model weights are accessible without copying. A symlink created at sidecar startup puts model data at a fixed path. The tradeoff is reduced isolation: every sidecar can see every other container’s filesystem. For clusters below Kubernetes 1.35, it remains a practical option. For 1.35 and above, image volumes replace it entirely.
LLM-Aware Request Routing via the Gateway API Inference Extension
Standard Kubernetes routing assumes equal request cost and distributes traffic evenly. That breaks with LLMs: a request producing 200 tokens takes hundreds of milliseconds while a thinking-model request can run for minutes. Both hit the same /completions path. The Gateway API inference extension solves this with an endpoint picker that continuously scrapes each model server and tracks in-flight requests, queue depth, and key-value cache state. It routes new requests to the instance with the best cache prefix match, reducing redundant prefill work. LMD extends this further with priority-based and capability-based routing.
RAG Pipeline Composition Using Kubernetes Primitives
RAG has two phases: ingestion (chunking documents and loading embeddings into a vector database) and query (retrieving similar documents and injecting them into the prompt). Every component in that pipeline is a standard Kubernetes workload. The vector database is a stateful service. The ingestion job is a batch job. The embedding model can run in-process, as a sidecar, or as a separate pod with independent scaling. No new primitives are required. The full pattern composes from building blocks the books already describe, which is why RAG maps so cleanly onto Kubernetes.
Notable Quotes
if you have to download 50 gigabytes of model data for every pot start. Of course this is does not scale right. Roland Huss · ▶ 12:10
image volumes is really the endgame and actually if you are able to use image volumes please use that Roland Huss · ▶ 19:14
the TLDDR is that this request routing moves from the request into the LLM state Bilgin Ibryam · ▶ 25:23
I would argue that uh rack is very very well suited for to be run within Kubernetes because we have already the primitives that support everything of those Roland Huss · ▶ 27:14
Key Takeaways
- KServe implements the controller pattern for model serving, abstracting GPU scheduling and runtime wiring.
- Image volume types in Kubernetes 1.35 eliminate the emptyDir copy step for 50 GB+ model weights.
- The Gateway API inference extension routes LLM traffic based on server state, not request shape.
About the Speaker(s)
Roland Huss is a Distinguished Engineer and Architect at Red Hat with over 25 years of programming experience. He currently focuses on integrating the Llama Stack within Red Hat AI and is co-author of the Kubernetes Patterns book.
Bilgin Ibryam is a product manager at Diagrid, where his team maintains the Dapr project, a CNCF graduated project for building durable and resilient agents. Previously an architect at Red Hat and Apache Software Foundation committer, he co-authored Kubernetes Patterns and writes about emerging tech trends.