AI
What an AI Gateway Actually Does
Traditional gateways route on headers and URL paths. AI inference requests carry the model name, prompt, and parameters in the body, so header-only routing misses the signal. The working group defines an AI gateway as a standard gateway extended to apply policies and routing decisions against the full payload, both request and response. Target use cases include token-based rate limiting, fine-grained access control, semantic caching, semantic routing, guardrails, and API key injection.
Payload Processing: Body-Aware Policy in Order
Gateway API today operates on headers and URLs, not the body. Shachar Tal’s proposal adds a declarative processor chain to HTTP route rules. Validation-only processors can run concurrently to reduce latency. Mutating processors must run in sequence because each mutation changes the input to the next processor. Each processor carries a timeout and a failure mode, letting operators decide whether a downed PII scanner fails open or closed. Processors can also short-circuit a request entirely, returning a cached response or blocking a disallowed prompt.
Ordering Matters: Jailbreak Detection Depends on Model Selection
Gateway API filters carry no ordering guarantees, which breaks payload processing. In semantic routing, a filter selects the target model first. A jailbreak detection guardrail that runs afterward must know which model was chosen, because jailbreak prompts differ across model families. Without a defined sequence, the guardrail fires against the wrong model. The working group’s processor chain enforces explicit execution order, solving the class of problems where mutation in one step changes what a later step must evaluate.
The Egress Gap: No Consumer-Focused Primitives Today
Gateway API was built for ingress from the producer’s perspective. Egress today requires a Service, an ExternalName CNAME, and a BackendTLSPolicy stacked together, where TLS policy describes how things connect inbound rather than how the client connects out. Istio’s ServiceEntry, Linkerd’s egress resource, and others each invented their own fix, producing fragmentation across vendors. The working group treats that fragmentation as a signal that Gateway API needs new consumer-focused primitives rather than workarounds on top of Service.
The Backend Resource: Default Policy Without Copying YAML
The proposed backend resource gives multiple HTTP routes a single target they can all reference. Attaching credential injection or circuit-breaker policy to the backend sets a default that applies across every route pointing at it. More specific routes can override or extend that policy without duplicating configuration. The resource also unifies internal and external inference providers under one surface, so a cluster that exhausts its self-hosted LLM capacity can fall back to an external provider while reusing the same policy attachment.
Demo: Egress Gateway Prototype with Inline TLS
A working prototype ran on a laptop the night before the talk. It shows an egress gateway resource wrapping a plain Gateway, a backend defined by destination hostname for httpbin, and inline TLS policy attached directly to that backend. An HTTP route targets the backend. A curl from inside the cluster sends plain HTTP, and the gateway initiates HTTPS on the way out, handling TLS origination transparently. Multiple namespaces reuse the same TLS configuration via reference grants, removing the cross-namespace TLS export pain present today.
Notable Quotes
For a growing class of workloads, the body is where the signal is. Shachar Tal · ▶ 08:55
there’s no good way to have default policy for a backend today unless you use something that’s vendor specific. Morgan Foster · ▶ 21:41
If your PII scanner is down, do you fail open or fail closed? Shachar Tal · ▶ 10:49
We have fragmentation across vendors trying to solve a problem that is a signal to us that we need new primitives in gateway API to support these use cases. Morgan Foster · ▶ 17:11
Key Takeaways
- AI inference routing requires body inspection, not just header matching.
- Mutating payload processors must run in sequence; validators can run concurrently to cut latency.
- Vendor fragmentation on egress (Istio, Linkerd, others) signals a missing Gateway API primitive.
About the Speaker(s)
Nir Rozenbaum is a Maintainer of Kubernetes Inference Gateway and Chair of the Kubernetes AI Gateway Working Group at Red Hat. He has a deep track record in CNCF open source, with a focus on AI/ML workloads on Kubernetes and shaping inference infrastructure standards.
Morgan Foster works on distributed inference and platform primitives for agentic systems at Red Hat. She is a maintainer of the Kagenti project, a framework-neutral middleware for managing agent workloads on Kubernetes, and co-authored the working group’s egress proposal.
Shachar Tal is a Distinguished Engineer at Palo Alto Networks focused on runtime security and Gateway API security for AI use cases. He authored the working group’s payload processing proposal and previously worked at Cisco and NVIDIA on NFV, CNIs, and KubeVirt.