The Cost of Running AI Locally

▶ Watch (00:40)

Running a model locally costs nothing and keeps data on the device. No cloud bills, no network dependency, no quota limits.

“You have full control over that data.” — Speaker 1

The problems start at hardware. Models are large, and devices vary. One user has a good GPU; another has a CPU-only laptop. Developers targeting millions of users cannot manually track which model variant fits which machine, or decide at runtime whether the CPU, GPU, or NPU path is the right one.

What Foundry Local Does

▶ Watch (02:11)

Foundry Local runs as a service on the device. The developer requests a model by family name, such as Qwen 2.5, and Foundry Local selects the right quantized variant for the hardware available, whether that is a CPU, GPU, or NPU build.

“You can’t run any model on any hardware.” — Speaker 1

The service spins up an OpenAI-compatible HTTP endpoint. Code written against the OpenAI SDK or Microsoft Extensions AI calls that local endpoint without modification. In the C# SDK, three lines cover it: import the namespace, name the model, start a FoundryManager, get a client.

Separating Orchestration from Consumption with .NET Aspire

▶ Watch (03:37)

Without Aspire, a single application handles both model download and inference calls. That mixes concerns: the app manages its own dependency before it can do its actual job.

“With Aspire, we separate that out.” — Eric

The Aspire app host owns orchestration. It downloads the model, starts the Foundry Local service, and holds the web application in a waiting state until the model is ready. The web application only consumes. It gets a chat completions client injected, converts it to a Microsoft Extensions AI interface, adds function calling and OpenTelemetry, and starts serving requests.

Live Demo: Wiring It All Together

▶ Watch (04:48)

Starting from a Microsoft Extensions AI template in Visual Studio, the presenter adds two NuGet packages: the Foundry hosting integration to the app host and the Aspire Azure AI inference package to the web project. Four lines in the app host define a Foundry resource, attach a Qwen 2.5-0.5B chat model, pass a reference to the web project, and declare waitFor on that model.

The Aspire dashboard model download live (09:06) showed the roughly 800 MB Qwen model downloading while the web app held in waiting state. Foundry Local selected the CPU build automatically because the demo laptop had no GPU or NPU.

Notable Quotes

You have full control over that data. Speaker 1 · ▶ 1:00

You can’t run any model on any hardware. Speaker 1 · ▶ 1:28

With Aspire, we separate that out. Eric · ▶ 4:09

Key Takeaways

  • Foundry Local picks CPU, GPU, or NPU model variants automatically based on the device it runs on.
  • The Aspire app host holds the web application in a waiting state until the model finishes downloading.
  • The endpoint Foundry Local exposes is OpenAI-compatible, so existing SDK code works without changes.