Running Local Models Without the GPU Container Problem

▶ Watch (00:24)

Models need GPUs, and running GPU workloads inside containers during local development is hard. Docker Model Runner solves this by pulling models from Docker Hub or any OCI-compatible registry and running them natively on the host, not inside a container. Full GPU access. OpenAI API-compatible endpoints at localhost:12434.

The docker model pull and run CLI demo (02:02) shows the full loop: pull phi4, list cached models, fire a prompt. Responses are fast because the model stays loaded. Docker Hub already carries Gemma, Llama, Phi, Mistral, and Mixedbread; docker model package lets teams push their own.

Accessing Local Models from Any App or Compose Stack

▶ Watch (06:43)

Any app using OpenAI client libraries connects to Model Runner by swapping one base URL. From the host: localhost:12434. From inside a container: model-runner.docker.internal. The .NET app streaming from local model (05:54) shows this: a .NET app built on the OpenAI SDK streams a response from phi4 with no other code changes.

Docker Compose picks up a model provider plugin. Declare the model as a dependency and docker compose up pulls it before the app starts. The Docker Compose sentiment analysis app (07:36) runs phi4 for sentiment classification and a second embeddings model for clustering, all from one compose file.

What MCP Actually Does and Why Setup Is Still a Problem

▶ Watch (10:20)

When an app sends a request to an LLM, it can include tool descriptions. The LLM reads them and, if it needs one, returns a tool call instead of a plain answer. The app runs the tool and feeds the result back. MCP standardizes this pattern: connect to an MCP server and its tools become available to any compatible client.

The problem is runtime friction. GitHub’s MCP server needs npx. A Python-based server needs uvx. Each new server pulls in a different dependency. Developers must install all of them manually before any tool works in their editor.

Containerized MCP Servers and Central Credential Management

▶ Watch (14:42)

MCP Catalog packages MCP servers as containers. No Node, no Python, no Go needed on the host. Containers also tighten security: when a containerized MCP server runs, it can’t touch the host file system unless explicitly granted access.

MCP Toolkit acts as a local gateway. Set API tokens once in Docker Desktop, enable or disable servers there, and every AI client reads from the same place. The MCP Toolkit + Copilot GitHub demo (17:16) shows 32 tools appear in VS Code Copilot the moment GitHub is enabled, drop to two when toggled off, and return when re-enabled.

Notable Quotes

model-runner.docker.internal Michael Irwin · ▶ 5:24

isn’t that what containers solved Michael Irwin · ▶ 14:36

All I know is it’s a container and I can just run it. Michael Irwin · ▶ 15:02

Key Takeaways

  • Docker Model Runner runs LLMs natively on the host with full GPU access, exposed through an OpenAI-compatible API at localhost:12434.
  • Docker Compose gains a model provider plugin so any app can declare its LLM dependency alongside its database and spin up in one command.
  • Docker MCP Toolkit acts as a central gateway for containerized MCP servers, managing credentials and tool availability across all AI clients at once.