AI
Why Traditional SLIs Break for LLM Workloads
SREs have spent 15 years optimizing stateless services. LLMs are stateful. Each model instance holds a KV cache, a key-value store built from previous prompts, that carries context between requests. A standard REST latency check misses this entirely. Non-deterministic outputs mean the same prompt can produce different responses, so pass/fail error rates tell only part of the story. The monitoring tooling that works for databases and microservices needs rethinking before it applies to a fleet of inference servers.
The LLM Request Lifecycle and the Metrics That Matter
An LLM request has two distinct phases. Prefill is the pause after you submit a prompt, when the model builds its initial KV cache. Decode is when tokens stream back. These phases have different resource profiles: prefill is compute-heavy, decode is memory-heavy. Time to first token measures the prefill phase and is one of the most telling SLIs. If that number rises on a customer service chatbot, something is wrong with the system. Goodput, the end-to-end throughput quality across the full request lifecycle, captures what a point-in-time error rate cannot.
KV Cache Hit Rate and LLM-D’s Cache-Aware Routing
KV cache hit rate is the metric most directly tied to LLM-D. When a new prompt is 80% similar to a previous one, routing to the instance that already holds the relevant cache yields 100 to 200% improvement in time to first token. LLM-D handles that routing decision. Splitting prefill and decode into separate pods, rather than running them in one monolithic process, cuts latency a further 5 to 30% and avoids over-provisioning GPU. Each component becomes a first-class Kubernetes citizen with individually scrapable metrics.
Batching as a Cost and Performance Tool
Batching groups similar workloads so the GPU processes them together. It is not new, but it behaves differently for LLMs than for CPU-based data normalization jobs. In an LLM context, batching improves user-facing latency and reduces cost at the same time. Off-peak hours, when traffic is low, are natural windows for offline batch jobs such as reinforcement learning or fine-tuning runs. Tracking throughput through those windows as a separate SLO gives operations teams a signal that is distinct from interactive chat latency.
Agentic Systems: SLOs Become Cross-Domain
Agentic workloads make the SLI picture two to three times more complex. East-west communication between agents means telemetry must span multiple services, not just the model host. Guardrails execution, tracking whether an AI gateway fires correctly, becomes its own SLI. Hilliary Lipsig also advocates NLP output scoring: saving inputs and outputs from agentic runs and grading them for correctness. This mixes objective and subjective data. Security teams, AI teams, and application teams all become stakeholders, turning model reliability into a cross-domain problem.
Real Architecture: Zelda Bots as a Production Proxy
Nuland built a system where hundreds of Game Boy instances run Legend of Zelda: Oracle of Seasons simultaneously, orchestrated through Ray, load balanced by LLM-D, and streamed over WebSocket. Tens of thousands of data points feed into it in real time. When time to first token spiked unexpectedly, it revealed a failure in how the LLM was processing reinforcement feedback, something the Ray console alone did not surface. The same architecture, and the same SLI discipline, applies to robotics, medicine, and automotive manufacturing.
Q&A
How did the Zelda speedrunning experiment connect to the GDC presentation? The system Nuland built for KubeCon is the same one he presented at the Game Developer Conference, where he demonstrated reinforcement learning on Game Boy emulators; a prior Double Dragon run produced a technique already known to the speedrunning community. ▶ 23:50
Will SLMs or larger mixture-of-experts models dominate inference fleets? Lipsig argues SLMs win on cost and training overhead; Nuland expects larger mixture-of-experts models, with SLMs still playing roles in guardrails and semantic routing. Both called it a coin flip. ▶ 26:22
Notable Quotes
we’ve been pushing stateless for about what 15 years now. And now we actually have to go back and start looking at how things are very stateful when we start talking about AI. Christopher Nuland · ▶ 04:23
we can have upwards of 100 to 200% increase in time the first token by making sure that we’re routing to the correct cache Christopher Nuland · ▶ 08:35
I saw time to first token shooting up really really high and it told me something was wrong in the way that the LLM was processing some of the the reinforcement feedback. Christopher Nuland · ▶ 22:02
Catastrophic forgetting. When you fine-tune your LLM and it suddenly doesn’t know anything anymore. That is called catastrophic forgetting. Hilliary Lipsig · ▶ 26:49
Key Takeaways
- KV cache hit rate is the SLI most directly improved by LLM-D’s cache-aware routing, with 100 to 200% time-to-first-token gains.
- Splitting prefill and decode into separate pods cuts latency 5 to 30% and prevents GPU over-provisioning.
- Agentic workloads require guardrails execution and NLP output scoring as new SLI categories across security, AI, and app teams.
About the Speaker(s)
Christopher Nuland is a Principal Technical Marketing Manager for AI at Red Hat, where he has worked for over six years. He focuses on machine learning and big data analytics, primarily for finance and agriculture customers, and leads Red Hat’s involvement in the vLLM and LLM-D communities.
Hilliary Lipsig is a Senior Principal Site Reliability Engineer at Red Hat, where she works primarily on Azure Red Hat OpenShift. A self-taught generalist with a background in QA engineering, she co-authored a book on AI ops alongside CNCF ambassador Andreas Grabner and reliability engineer Rob Robie.