The Production Ceiling: Session Affinity, Policy, and Observability

▶ Watch (0:43)

MCP traffic hits a production ceiling at scale. Three problems emerge. Session affinity: MCP’s stateless protocol conflicts with load-balanced deployments. Policy enforcement: JSON RPC in HTTP body hides tool names, making authorization impossible at the gateway. Observability: OpenTelemetry for HTTP does not cover MCP attributes. A fourth problem: migrating existing REST APIs to MCP requires bridging without rewriting services. Google encountered these when connecting agents to internal and third-party MCP servers.

Envoy’s Three Filters for MCP Traffic

▶ Watch (8:03)

Three Envoy filters handle MCP traffic without new infrastructure. The MCP filter unboxes JSON RPC bodies, extracts tool names, and enables RBAC, mutual TLS, and access logs using Envoy’s existing architecture. The MCP to REST bridge converts any REST API into an MCP-aware endpoint without code changes. Google uses it for Maps and Spanner. The MCP router aggregates multiple MCP servers behind a single URL. These filters are open source in Envoy.

The Road Ahead: Streams, Async, and Service Discovery

▶ Watch (17:10)

Future work includes stream transports (WebSocket and gRPC) and async task support for long-running MCP operations. Service discovery merges control plane and data plane: a registry for agents, MCP servers, and tools. Envoy AI Gateway runs standalone or in Kubernetes, providing horizontal scalability without losing session context. The community has nearly 50 maintainers. The goal is to make Envoy agent-ready everywhere.

Notable Quotes

We hit this thing that I like to call the production ceiling. Erica Hughberg · ▶ Watch (0:43)

right now we have the MCP filter to make Envoy MCP aware. Boteng Yao · ▶ Watch (8:03)

Ultimately, we have these three filters that are all here to help Envoy understand MCP traffic. Erica Hughberg · ▶ Watch (12:21)

uh we also have the MCP to REST bridge. Boteng Yao · ▶ Watch (10:09)

all of you in here are Envoy users, whether you like it or not. Erica Hughberg · ▶ Watch (22:52)

Key Takeaways

  • Session affinity, policy enforcement, and observability are the three MCP scale problems.
  • Envoy’s MCP, REST bridge, and router filters solve them without new infrastructure.
  • Future work includes stream transports, async tasks, and an agent registry.

About the Speaker(s)

Erica Hughberg is a technical leader, software engineer, and community advocate passionate about helping engineering teams develop scalable, secure, and user-focused application platforms. As a maintainer of Envoy AI Gateway, she concentrates on features that enable organizations.

Boteng Yao is a Senior Envoy Maintainer and Software Engineer at Google, working on Envoy for various products with an emphasis on data plane, reliability, and security.