AI
Ten Years of Envoy: From Lyft’s Internal Proxy to Cloud-Native Foundation
Envoy started 10 years ago as an internal proxy at Lyft. It now sits inside service meshes, API gateways, and edge proxies across the industry. The project has three active sub-projects: core Envoy proxy, a high-performance L7 proxy that has been the foundation throughout; Envoy Gateway, a Kubernetes-native control plane built on the Gateway API; and Envoy AI Gateway, a data and control plane extension targeting AI inference traffic.
Why AI Traffic Breaks Standard HTTP Assumptions
Streaming inference calls are long-lived connections, not short HTTP request-response pairs. Cost per request varies widely based on model size and token count. Agentic workflows compound this: one trigger spawns chains of tool calls across multiple backends. Standard endpoint latency metrics miss all of that. Operators need per-model visibility into token usage and cost, not just whether an endpoint is slow.
Roadmap: AI Features Moving Into the Core Proxy
The roadmap pulls AI traffic management features directly into core Envoy. Native protocol support for MCP and agent-to-agent communication is coming, along with request buffering and cross-service failover. Envoy Gateway gains new extensibility points and filter integrations. Envoy AI Gateway is graduating from alpha to beta, adds support for the OpenAI responses API standard, ships token budget and rate-limit policies, and provides richer telemetry for LLM backends.
Adoption Numbers Show Accelerating Growth
Envoy Gateway hit 2.1 million Helm chart pulls in a single 30-day window. That growth tracks with the NGINX ingress migration drawing operators toward alternatives. The project also records 5 million image pulls for the gateway and over 27,000 GitHub stars on the core proxy. The contributor base has grown to over 300 unique gateway contributors. Szwaj attributes part of the acceleration to that migration event pulling new teams toward Envoy Gateway.
Notable Quotes
streaming inference calls is long live connections not traditional HTTP request response
Karol Szwaj · ▶ 1:19
not just the endpoint is slow but we need to find out which model how many tokens it uses and at what cost
Karol Szwaj · ▶ 1:43
recently like gateway reach out 2.1 million hamchart pools over 30-day window
Karol Szwaj · ▶ 3:02
Key Takeaways
- Envoy AI Gateway moves from alpha to beta, adding token budgets and LLM telemetry.
- AI workloads require per-model, per-token observability that standard HTTP metrics cannot provide.
- Envoy Gateway reached 2.1 million Helm chart pulls in 30 days, with 300 contributors.
About the Speaker(s)
Karol Szwaj is a senior software engineer at Kubermatic and a maintainer of Envoy Gateway and kube-bind. He currently works on kcp and specializes in open-source technologies, distributed systems, and networking, with a telecommunications background from university.