AI
Why Tool Calling Breaks at Scale
Stuff a few tools into an LLM context and it works. Stuff in Google services, Jira, a wiki, and hundreds more, and the context blows up. Composition breaks. Every action becomes a round trip back to the model.
“tool calling gets weird at scale.” — Sunil Pai
Code mode skips the JSON back-and-forth entirely. The model writes JavaScript (a language it has trained on terabytes of) and runs it once against a live environment. Looping, state, parallelization: all of it happens inside that single execution, not spread across eight model calls.
Two Tools Instead of 2,600
Cloudflare’s API surface has 2,600 endpoints. Exposing a tool per endpoint costs 1.2 million tokens on the first call. Matt Carey solved this with two tool calls: search and execute. Both take code as input. That collapsed the context from 1.2 million to 1,000 tokens, a 99.9% reduction.
The Cloudflare MCP live demo (05:28) shows the model handling a DDoS response request in one shot, versus eight round trips through a conventional MCP server.
Inhabiting the State Machine
Kenton Varda drew a tic-tac-toe board on a canvas app and told Claude Opus to play. The model tried to generate a tic-tac-toe app. Kenton stopped it: the entire system state was already there as an array of strokes. He said inspect that and play.
“To be clear, there is no tic-tac-toe” — Sunil Pai
The model read the strokes, recognized the board, and drew a circle in the center. No game code. Pai calls this inhabiting the state machine. The reasoning traces later showed Opus deliberately let Kenton win.
The Harness Architecture
The pattern everyone is building is a harness: a sandbox that starts with zero capabilities, then gets APIs granted explicitly. No outgoing fetches by default. Cloudflare uses V8 isolates (fast startup, ten years of security hardening) but the runtime choice is secondary.
“Absolute observability on these systems.” — Sunil Pai
From there: long-running workflows that carry state across months, per-user generative UIs, and agent-friendly APIs backed by markdown docs. The agents interacting with your systems live in registries and reason in types, not chat boxes.
Notable Quotes
tool calling gets weird at scale. Sunil Pai · ▶ 1:35
LLMs are breaking this boundary. Sunil Pai · ▶ 8:39
To be clear, there is no tic-tac-toe Sunil Pai · ▶ 10:44
Absolute observability on these systems. Sunil Pai · ▶ 13:56
Key Takeaways
- Replace hundreds of JSON tool calls with generated code that runs in a single execution.
- Two endpoints — search and execute — can cover a 2,600-endpoint API surface with 99.9% fewer tokens.
- A capability-based sandbox starts empty, grants APIs explicitly, and logs every decision for full auditability.