Air-Gapped Use Cases for Local LLMs

▶ Watch (00:13)

Air-gapped environments don’t have a cloud fallback. Diaz Conscha works on three projects where that constraint is real: a telemedicine system inside prisons, copper and iron mining sites with no connectivity, and retail kiosks that need anonymous customer analytics. Each case needs inference on the machine. That’s the opening premise for Foundry Local, a Microsoft tool announced two days before this session that downloads and runs models entirely on your hardware.

Installing Foundry Local and Managing Models

▶ Watch (01:49)

Foundry Local installs via winget on Windows or brew on Mac. The foundry model list command shows all available models: Phi, DeepSeek at 14B and 7B, and others.

“And most importantly, they have different versions depending on the hardware where you’re running Foundry local.” — Rodrigo Diaz Conscha

When Diaz Conscha downloaded Phi 3.5, Foundry automatically selected the CPU variant for his machine. He had to pass the model ID explicitly to force the GPU version. Both sit under ~/.foundry/cache/models, organized by model author. Custom ONNX-format models load the same way.

GPU vs CPU Inference in Practice

▶ Watch (05:34)

Foundry Local exposes every loaded model as a chat completions endpoint on port 5273. Diaz Conscha ran the GPU vs CPU live comparison demo (07:07), querying the GPU variant of Phi 3.5 via Postman, then opening Task Manager to show the GPU utilization spike. Sending the same prompt to the CPU variant pegged the processor instead. Multiple models, GPU and CPU, can run simultaneously, constrained only by available hardware. For kiosks with no GPU, distilled ONNX models are the practical workaround.

Integrating Local Models into Applications

▶ Watch (11:44)

Once a model runs as a local endpoint, any OpenAI-compatible framework connects to it without code changes. Diaz Conscha wired up a .NET console app using Microsoft Semantic Kernel, pointing AddOpenAIChatCompletion at localhost:5273. The app answered prompts from the local model. Then he disconnected Wi-Fi to make it concrete.

“There’s no Internet connectivity right now.” — Rodrigo Diaz Conscha

Responses still came through. LangChain, LangGraph, and Semantic Kernel all work against the same endpoint, making the swap from a hosted model largely transparent to application code.

Notable Quotes

Local large language models running in your machine. Rodrigo Diaz Conscha · ▶ 15:08

And most importantly, they have different versions depending on the hardware where you’re running Foundry local. Rodrigo Diaz Conscha · ▶ 4:09

There’s no Internet connectivity right now. Rodrigo Diaz Conscha · ▶ 14:53

Key Takeaways

  • Foundry Local runs on Windows or Mac via a single CLI command and exposes models on a local HTTP endpoint.
  • GPU variants respond faster; CPU-only hardware like retail kiosks must use distilled ONNX models instead.
  • Any OpenAI-compatible framework, including Semantic Kernel and LangChain, works against the local endpoint without code changes.