AI
Why Token Asymmetry Matters More Than Air Gaps
Anthropic’s training clusters take in tokens (text, images, tool calls) and output tokens. The model weights never need to leave. Debugging, metrics, and control plane traffic are measured in megabytes per second. Weights are terabytes and mostly incompressible. A true air gap would require shipping researchers into a data center. That is not happening. Instead, the team aimed to make exfiltration prohibitively slow by exploiting that size asymmetry.
Building a Dumb Perimeter Appliance
The core idea: force all traffic through a single egress point and drop packets when a byte quota is exceeded. The appliance counts every byte, including TCP acknowledgements, DNS, and keepalives. It uses a token bucket filter that accumulates bandwidth during idle periods and allows bursts. A per-second rate limit would break researcher workflows. The bucket can hold several hours of total bandwidth, so a researcher pushing research results sees gradual slowdowns, not hard blocks.
The Rollout: 98% Egress Reduction and Painful Lessons
Deploying Gatekeeper was a nightmare. Services did not fail immediately; they slowed down and then timed out, causing cascading failures that were hard to trace. The team lacked observability tools for gradual degradation. At one point, 30% of all networking tickets included the question “Is it Gatekeeper?” After four to six months of reclassifying buckets and migrating services, egress dropped by 98%. The remaining 2% includes essentials like SSH, metrics, and hosted pages that cannot be eliminated easily.
Why a Perimeter Control Is a Fallback, Not a Solution
Perimeter controls are a concession when you cannot fully secure the inside. The real goal is confidential compute that minimizes which software touches unencrypted weights. That takes years and may require new hardware. Until then, Gatekeeper buys time. Accidental exfiltration attempts now take hours or days and trigger alerts. The same platform also enables TLS interception and domain allow lists.
Q&A
How does this apply to inference clusters that must send legitimate token traffic? Anthropic applies the same rate limiter to inference clusters but exempts customer token flows by hard coding those egress paths as accepted risk. ▶ 29:15
Is DDoS from the ingress side a significant threat given the focus on egress? The architecture forced every path into the cluster to be locked down, leaving only a few endpoints reachable from outside. Tiered enforcement would isolate any saturation to a single bucket or node. ▶ 30:48
Has Anthropic done much work on confidential compute? The speaker noted confidential compute is still “full of holes” but values its second-order effects: minimizing the trusted computing base around sensitive assets. Anthropic is working with hardware vendors to improve offerings. ▶ 32:07
Notable Quotes
“We spent a bunch of effort trying to make sure that even a full compromise of our compute clusters does not allow a pivot into this kind of like egress limiter environment so that someone could like turn that thing off.” Ziyad Edher · ▶ 13:37
“In the end, we did manage to kill off about 98% of all of our egress, which was huge.” Ziyad Edher · ▶ 23:04
“This isn’t about shipping a hundred researchers into a data center in the middle of nowhere. The fact of the matter is you can’t properly air gap a research cluster and still do a bunch of remote research.” Ziyad Edher · ▶ 0:46
Key Takeaways
- Model weight incompressibility creates a physical constraint that attackers cannot bypass with software tricks.
- A token bucket rate limiter enables burst traffic while capping total exfiltration to days or weeks.
- The rollout exposed hidden dependencies and forced teams to lock down every network path into the cluster.