Running AI Models Locally Without API Keys

▶ Watch (00:15)

Foundry Local shipped that morning. It runs AI models on device using ONNX Runtime, which means no API key, no cloud subscription, and nothing sent off the machine. Hardware requirements are modest: 8 GB RAM minimum, 16 preferred, plus 3-11 GB per model. The foundry model list CLI demo (02:24) shows CPU and GPU variants side by side. Pass the short alias like phi-4 and Foundry picks the best variant for your hardware automatically.

“So just to reiterate, this is all running completely offline.” — Natalie

Scaffolding a Flask App with an On-Device Model

▶ Watch (04:38)

DeepSeek-R1 7B generates the full Flask app from one prompt: a single-page frontend with a city picker, a backend route calling an OpenAI-compatible chat completions endpoint, and a README with run instructions. The whole output arrives in seconds. Pointing the app at Foundry Local is one URL substitution: set FOUNDRY_SERVICE_URI to the address from foundry service status, then swap that into the base_url field. The travel app live in browser (10:23) shows a Sydney itinerary returned offline in under two seconds.

“Just a one line change to add the service Uri” — Autumn

Replacing the REST API with the Foundry Local SDK

▶ Watch (11:38)

The REST API requires copying the full model variant ID from foundry model list, which is tedious. The Python SDK fixes that.

“It’s pip installed Foundry local SDK for Python.” — Autumn

Import FoundryLocalManager, call it to get the service URI, and manual endpoint management disappears. The app also gains a model picker: select any catalog entry and the SDK downloads it if not already cached, then runs immediately. JavaScript and C# SDKs are in development.

Notable Quotes

So just to reiterate, this is all running completely offline. Natalie · ▶ 3:55

Just a one line change to add the service Uri Autumn · ▶ 9:35

It’s pip installed Foundry local SDK for Python. Autumn · ▶ 11:54

Key Takeaways

  • Foundry Local runs ONNX models on Mac and Windows with no API keys and no internet connection required.
  • Swap the OpenAI endpoint URL for the local Foundry service URI to migrate existing apps in one line.
  • The Python SDK replaces manual endpoint management and adds in-app model catalog browsing.