The Hardware and Model Shift That Made Local AI Viable

▶ Watch (00:19)

60 years of Moore’s Law left hardware ready for local AI. Copilot+ PCs now ship with NPUs, ONNX Runtime gets 10 million downloads a month and runs 4x faster than raw PyTorch in internal tests, and models like DeepSeek run on consumer hardware.

Customer demand, not technology, drove the product decision. Liquid Text needs sensitive legal documents to stay on-device. Moises Live needs low-latency audio processing. A game developer facing hundreds of millions of daily inference calls needs cost control. Distribution centers in remote Africa and Asia need AI without reliable connectivity.

What Foundry Local Is and How It Is Built

▶ Watch (04:39)

Foundry Local stacks three layers. ONNX Runtime sits at the bottom and accelerates across NVIDIA CUDA, AMD, Intel, Qualcomm, and Apple Silicon. The Foundry Local management service hosts models on-device and pulls from a catalog of 1,900+ open-source models via Azure AI Foundry. On top, a CLI and two SDKs handle app code and service management.

The Inference SDK is the same one used against Azure cloud endpoints. Changing from cloud to local is a configuration switch, not a rewrite. The Management SDK, available in Python and JavaScript, starts the service and loads models from within your application.

Migrating a Cloud App to Local With Two Lines of Code

▶ Watch (11:03)

Mo Tang walked through the two-line code swap demo (12:27) on a Windows machine. The existing app pointed at an Azure AI Foundry cloud endpoint with a GPT-4o mini model. To switch it local, he typed “local” into the endpoint field, called foundry_sdk_init() to start the service and load the model, and assigned the local endpoint and model name. Everything else stayed the same.

“you literally have no code change.” — Mo Tang

The CLI demo followed. foundry model list shows silicon-specific variants for each model. CUDA, integrated GPU, and CPU builds all appear. On load, Foundry Local picks the best variant automatically.

KPMG’s On-Device Tax Document Processing

▶ Watch (19:14)

KPMG’s tax group processes hundreds of PDF documents per engagement. Clients are risk-averse and unwilling to send raw data to the cloud, even through Azure. Gaurav Sehgal and Sai Lori built a desktop utility that clients install locally: drag-and-drop a property tax bill, the on-device LLM (Phi-4 mini) parses it, produces a JSON object, and drafts an Outlook email with only the extracted fields, not the source document.

“So data protection and client privacy is paramount at KPMG.” — Sai Lori

The roadmap includes connecting the tool to KPMG’s Digital Gateway platform and running fine-tuned, tax-specific small language models on client machines.

Local Agents and Windows Actions

▶ Watch (29:46)

Foundry Local’s agent mode ships four catalog agents (file organizer, OCR, time, web search) alongside MCP servers. Each agent bundles a model and one or more MCP servers. foundry agent install downloads both. Kunal Vaishnavi ran the OCR agent receipt demo (32:58): the agent searched a downloads folder, ran OCR on a cafe receipt, and returned a total of $8.39 without being told the filename.

The background remover Windows actions demo (35:55) showed OS-level MCP servers at work. Before the Paint background-removal tool ran, a system pop-up requested user approval. Agent mode is in private preview; SDK integration is still in progress.

Notable Quotes

run exactly the apps completely local on this device, no cost, no Internet and fast and beautiful. Mo Tang · ▶ 11:14

you literally have no code change. Mo Tang · ▶ 12:21

So data protection and client privacy is paramount at KPMG. Sai Lori · ▶ 21:20

With just a few commands, I was able to get the mall running on my machine and within 24 hours we could start prototyping applications, solving real world scenarios. Sai Lori · ▶ 21:43

Key Takeaways

  • Foundry Local runs on ONNX Runtime and auto-selects the best silicon variant (CUDA, integrated GPU, CPU, Apple Silicon) at model load time.
  • Switching from Azure cloud to Foundry Local requires only two SDK lines; the rest of the Azure Inference SDK code stays the same.
  • Agent mode (private preview) exposes MCP servers locally and requires explicit user permission before any Windows action runs.