Windows ML as a Managed Runtime for Intel AI PCs

▶ Watch (05:44)

Building an AI app on Windows today means bundling a runtime, hardware plug-ins for every chip variant, and owning every update. Windows ML 2.0 erases that. Attach to its NuGet package and the OpenVINO execution provider pulls down automatically. When Intel ships an OpenVINO update for new silicon, Windows ML updates at the same moment, because OpenVINO is now a platform component.

Powder, a 15-person company that turns Copilot+ PCs into gaming highlight reels, had a prototype integrated in under a week.

“It saved them upwards of about 70% of the time” — Ryan Demopoulos

Targeting CPU, GPU, and NPU with One Line of Code

▶ Watch (14:16)

Intel Core Ultra machines carry three compute units: a CPU with AVX/VNNI, an Arc GPU with XMX matrix engines, and a dedicated NPU. Switching between them in Windows ML takes one parameter change in the session options. Freddy Chiu walked through the ResNet50 inference switching from CPU to NPU to GPU (13:48) live, showing CPU utilization fall to zero once the NPU session started. FP16 and INT8-quantized models both run across all three compute units without format changes, because Intel enforces data-type consistency across the platform.

Model Optimization: Foundry Local, Olive, and the NNCF Pipeline

▶ Watch (17:32)

Microsoft’s Olive toolchain now embeds Intel’s OpenVINO Neural Network Compression Framework. A single olive command pulls a model from Hugging Face, quantizes it for Intel NPU, and outputs an ONNX file ready for Windows ML. Foundry Local adds a catalog of pre-optimized models for developers skipping the pipeline. Freddy Chiu followed the optimization demo with the dual-LLM demo running two 7B models on GPU and NPU simultaneously (21:55), on a laptop in airplane mode on battery, with workloads shifted between processors by changing one line of code.

Local AI Agents and FastDraft Speculative Decoding

▶ Watch (28:19)

Guy Boudoukh ran Phi-4-mini on Intel Core Ultra with MCP-connected web search, YouTube transcript access, and a local PowerPoint builder. In the Phi-4-mini agent building a presentation via MCP search (29:59), the agent retrieved results and assembled slides in seconds, with a local embedding model handling transcript context via vector search.

Reasoning models generate more tokens per task, which strains on-device memory bandwidth. FastDraft addresses this with a draft model trained in under 48 hours on one GPU node. The FastDraft 2x speedup comparison (38:47) showed 2x average throughput gain and 3x bandwidth reduction versus 4-bit quantization alone. The Phi-4-mini draft model is open-source.

Intel on Azure: AMX, QAT, and Federated Learning

▶ Watch (41:20)

Brian Rogers argued that smaller agentic models running on Xeon AMX extensions shift matrix multiplications onto CPU, reducing GPU pressure for sub-11-billion-parameter workloads in Azure. Intel QuickAssist Technology (QAT) on Sapphire Rapids and Emerald Rapids Xeons offloads TLS, compression, and decompression to dedicated silicon. Microsoft’s Resilient File System now uses QAT-backed compression alongside an existing QAT integration with SQL Server backup.

“There is hardware, and hardware still matters,” — Brian Rogers

OpenFL trains models against data that stays on-site, with HIPAA-compliant deployments already running for skin cancer detection.

Notable Quotes

It saved them upwards of about 70% of the time Ryan Demopoulos · ▶ 11:34

And also today, we introduced a new draft model for Phi-4-mini. Guy Boudoukh · ▶ 38:05

There is hardware, and hardware still matters, Brian Rogers · ▶ 52:30

Key Takeaways

  • Windows ML embeds OpenVINO as a platform component, removing runtime dependency management from app developers entirely.
  • FastDraft trains a compact draft model in under 48 hours on one GPU node and delivers 2x inference speedup on Intel AI PCs.
  • Intel AMX on Xeon lets sub-11-billion-parameter models run efficiently on CPU in Azure, reducing workload pressure on GPUs.