The Cost Gap Between Large and Small Models

▶ Watch (1:23)

Sending every query to GPT-5 costs $2.50 per million input tokens and $15 per million output tokens. Microsoft Phi-3, a small language model, handles the same simple translation at $0.03 per million input tokens and $0.52 per million output tokens. Both return the same answer. The price difference is 27x. For a Telefonica-scale telco running models across edge, core data center, and cloud, that gap compounds quickly across millions of requests.

Why Agent Gateway Beats Envoy for This Problem

▶ Watch (5:08)

Envoy handles RESTful API traffic well: fail fast, stateless requests. LLM traffic is stateful and long-lived, which is a different shape of problem. Agent Gateway, a Linux Foundation project written in Rust, targets that pattern. The team reused the Envoy control-plane API, replaced the data plane with Agent Gateway, and plugged in a custom intelligent router as an extension. No new control-plane API to learn; the swap is transparent to upstream configuration.

How the Agentic Control Plane Configures Itself

▶ Watch (13:14)

Registering a model means creating one LLM backend CRD. An orchestrator agent written in Python with ADK watches that CRD and fires a chain of workers: it creates the HTTP routes and Agent Gateway backend, triggers an MMLU Pro evaluation job (70 requests in the demo), calls an electricity cost agent for locally deployed models, and calls a model pricing agent for remote ones. The CRD is updated with the full status, latency baseline, and cost before any user traffic arrives.

Three Routing Strategies and the Scoring Formula

▶ Watch (15:48)

The intelligent router sets HTTP headers that tell Agent Gateway which backend to use. It scores each backend with a weighted formula. Configuring the system for accuracy weights accuracy at 80%, cost and latency at 10% each. Configuring for cost flips the weights: 80% cost, 10% each for accuracy and latency. The same pattern applies to latency mode. The routing decision adds about 50 milliseconds per request because the scoring SLM runs on CPU inside the cluster, adding no external API cost.

Open Problems: Telemetry, Geolocation, and Zero Trust

▶ Watch (16:42)

The team documented 50 routing use cases. Two stood out: thermal-aware scheduling, which redirects traffic away from overheating GPU racks, and exponential backoff quota enforcement, which routes heavy users to cheaper models when they approach their token quota. Both require fresh, distributed metrics, and that remains unsolved. Geolocation adds another layer: one region may run four models, another seven. Istio provides the zero-trust foundation for securing cross-region traffic between backends.

Q&A

Does the agentic orchestration itself add infrastructure cost? The orchestrator and worker agents run on SLMs inside the Kubernetes cluster, so the added cost is zero. ▶ 27:43

How much latency does the routing layer add? About 50 milliseconds per request, which is not significant against typical LLM reply times of two to five seconds. ▶ 29:13

Why not use the Kubernetes Inference API instead of Agent Gateway? The Kubernetes Inference API targets local cluster routing only. Cross-region and remote model traffic requires a wider solution, which the Inference API does not yet cover. ▶ 32:00

What is the advantage over BLM Semantic Router? Semantic Router bundles all logic together and lacked full CRD support when the project started. The team needed an agentic flow to configure the router dynamically, which required building their own solution. ▶ 32:54

Notable Quotes

the the agentic workflow runs on SLMs to make the uh decision of which is the the the best model. So actually it runs inside the Kubernetes cluster. So no cost at all. Antonio Berben · ▶ 27:43

Since the SLM that make the decision is running on on CPU, it’s pretty fast. It’s around 50 milliseconds per per request. Antonio Berben · ▶ 29:17

you cannot use right now inference API, Kubernetes inference API because it’s intended to be in local routing in local clusters. So if it’s remote, we need to just go wider Antonio Berben · ▶ 32:07

The different between both is around 27 times x cheaper the second way if you wrote the request to the SLM Felipe Vicens · ▶ 02:31

Key Takeaways

  • Routing simple queries to Phi-3 instead of GPT-5 cuts token cost by 27x with identical output quality.
  • Agent Gateway, written in Rust, handles stateful LLM traffic better than Envoy for agentic workloads.
  • A single LLM backend CRD triggers the full agentic setup: routing, evaluation, and cost scoring automatically.
  • An 80/10/10 weighted formula lets operators prioritize accuracy, cost, or latency per deployment mode.
  • The Kubernetes Inference API cannot route across regions; distributed telco deployments need a wider gateway.

About the Speaker(s)

Felipe Vicens is a Principal Architect for AI and Virtualization at Telefonica. With over a decade of experience in Telco Cloud architecture, he focuses on infrastructure that runs network functions and AI workloads across edge and cloud environments. He is an active open-source advocate specializing in Telco Cloud, Edge, and AI operations.

Antonio Berben is a Principal Solutions Engineer at Solo.io, where he helps teams adopt service mesh at scale. He organizes KCD Spain and CNCF Iberia and contributes to open-source projects. His background in software development keeps developer experience at the center of his work.