AI
The Case Against Renting Your AI
Most AI runs in data centers. A cybersecurity researcher Cheema knows got locked out of Claude, Gemini, and ChatGPT simultaneously while doing legitimate penetration testing. That’s a real problem when AI is no longer just a chat tool but infrastructure you depend on daily.
“well which is like not your weights not your brain.” — Alex Cheema (quoting Andrej Karpathy)
Cheema’s argument: closed-source providers will monetize access because that is how they make money. A few companies controlling the most capable models creates a power concentration with no check. Local inference is that check.
Inference Is Memory-Bound, Not Compute-Bound
Training is compute-bound. Inference is not. At batch size one, which is what every local user runs, the bottleneck is memory: how much you have, how fast you can move it, and how much energy that movement costs. Prefill (loading your prompt) is compute-bound but largely irrelevant locally because a good harness caches most of it. Decode generates tokens one at a time and is always memory-bandwidth-bound.
Cheema ran Qwen 3.5 on Apple silicon and found real throughput 50% below theoretical. Kernel fusion alone, fusing unnecessary separate kernel launches into one, recovered 30% of that gap.
The Intelligence-Per-Joule Curve
Stanford’s Hazy Research group tracks “intelligence per joule”: task accuracy divided by the energy to run it. That metric has improved 5x from hardware alone and another 3x from model efficiency over the past two years. Those gains compound. Cheema’s thesis: there is still 100x available across the full stack (kernels, harness, model architecture, hardware) because so little of this work has been done under an inference-first design constraint rather than a training-first one.
The MacBook M5 Max, available off the shelf, ships with 128 GB of unified memory and 614 GB/s of bandwidth.
Why Local Batching Closes the Cloud Gap
Cloud providers batch millions of users together, which is why their unit economics beat local. But three trends erode that advantage. First, multi-agent systems like Grok 4.2 run four or more simultaneous agents per request, pushing a single user’s effective batch size to eight or more. Second, test-time scaling (running a 1B model ten times and picking the best result) fills the same role. Third, continual learning, where weights update at inference time, would make every user’s model unique and unbatchable, potentially making local inference 10x cheaper relative to the cloud.
Combining Heterogeneous Hardware With Exo
Mac Studios have lots of memory but modest bandwidth (800 GB/s on the 512 GB model). An RTX 5090 carries 32 GB of VRAM at roughly 1.5 TB/s. Cheema’s answer: use both. Run prefill on the compute-dense device and decode on the memory-bandwidth-rich one.
Exo handles distribution automatically. Install the app on each machine, connect however you want, and it discovers peers and partitions the model. In the GLM 5.1 on four Mac Studios via Exo (01:08:08), RDMA over Thunderbolt 5 cut inter-Mac latency 100x, to single-digit microseconds. The Spark plus MacBook prefill-decode split demo (01:41:20) cut a 100 KB prompt from 7 seconds to 4.8 seconds end-to-end.
Q&A
Are people actually going to spend $5,000 on a local inference machine? Where is this headed for consumers? Not yet, but within 18 months Cheema expects $5,000 to buy near-frontier performance at usable speeds, driven by compounding improvements across kernels, harness, models, and hardware. ▶ 18:50
Even with 128 GB of memory a Mac runs inference much slower than an RTX 5090. How do you think about Mac versus Nvidia hardware? Both have complementary strengths and Cheema argues you want both: Mac for memory capacity and decode, GPU for compute density and prefill. ▶ 38:34
Have you used auto-research loops to optimize models for your hardware? Cheema is skeptical of undirected auto-research, calling it a slot machine; it needs a well-reasoned hypothesis and the scientific method to produce real results. ▶ 1:36:22
Notable Quotes
well which is like not your weights not your brain. Alex Cheema (quoting Andrej Karpathy) · ▶ 3:00
with RDMA, it’s 100 times faster. So, Alex Cheema · ▶ 1:10:56
run things locally. Uh that’s my thesis. Alex Cheema · ▶ 29:43
Key Takeaways
- Decode is memory-bandwidth-bound; local inference on consumer hardware is within 100x of optimal across the full stack.
- Combining a compute-dense GPU for prefill with a memory-rich Mac for decode delivers 2x speedups today on large prompts.
- Continual learning would break cloud batching entirely, making local inference 10x cheaper relative to the cloud if it lands.