Windows ML

Windows ML provides a hardware abstraction layer that automatically selects and utilizes the most appropriate compute device for a given model. It supports:

• DirectML: A low-level API for high-performance GPU acceleration.
• CPU Fallback: Reliable execution on the CPU if no compatible accelerator is found.
• Vendor Drivers: Integrates with drivers from Intel, AMD, NVIDIA, and Qualcomm to leverage their specific AI acceleration features (e.g., Intel DL Boost, NVIDIA Tensor Cores). This ensures optimal performance and power efficiency by matching model operations to the best available silicon.

The API is built around the Open Neural Network Exchange (ONNX) format as its primary model representation. This provides:

• Framework Agnosticism: Models trained in PyTorch, TensorFlow, Scikit-learn, or others can be exported to ONNX and run directly.
• Portability: A single ONNX model file can be deployed across the Windows ecosystem without modification.
• Graph Optimizations: Windows ML applies runtime optimizations like constant folding, node fusion, and layout transformation to the ONNX graph for faster execution on target hardware.

Windows ML offers flexible programming models to suit different application needs:

• Synchronous Evaluation: Blocking calls (Evaluate method) for simplicity in scenarios where UI responsiveness is not critical.
• Asynchronous Evaluation: Non-blocking calls (EvaluateAsync method) that prevent the UI thread from freezing during inference, essential for responsive desktop applications.
• Batch Inference Support: Allows processing multiple inputs in a single call to amortize overhead and improve throughput, particularly useful for GPU execution.

The API uses a binding mechanism to manage inputs and outputs, abstracting low-level tensor memory operations. Key features include:

• Tensor Abstraction: Data is passed via TensorFloat, TensorInt, etc., objects, handling data type conversion and layout (e.g., NHWC vs. NCHW).
• Direct Buffer Access: For maximum performance, advanced bindings allow mapping model inputs/outputs directly to pre-allocated GPU buffers (e.g., Direct3D textures).
• Software Bitmap Interop: Seamless conversion from SoftwareBitmap (common in UWP camera pipelines) to the tensor format required by vision models.

Windows ML is supported by developer tools within the broader Windows ecosystem:

• Windows ML Dashboard: A system settings page that shows which applications are using ML and on which hardware (GPU, CPU, etc.).
• Model Analysis with Windows Performance Analyzer: Allows deep profiling of inference sessions to identify performance bottlenecks in the model graph or hardware dispatch.
• WinMLRunner Tool: A command-line utility for validating model loading and execution, and benchmarking performance on local hardware.

The API simplifies the deployment of AI features within Windows applications:

• Side-Loading: Models can be packaged and distributed directly within an application's bundle, avoiding cloud dependency.
• Model Versioning: Applications can manage multiple model versions locally and select the appropriate one at runtime.
• Windows Update Integration: Critical runtime components for Windows ML are serviced via Windows Update, ensuring security and compatibility fixes are delivered automatically.

The open, interoperable format for representing machine learning models, which is the primary model format for Windows ML. An ONNX model encapsulates the computational graph, operators, and trained parameters.

• Framework Agnostic: Models trained in PyTorch, TensorFlow, or scikit-learn can be exported to ONNX for use in Windows ML.
• Portability: The format ensures the model's architecture and weights are preserved across different runtimes and hardware platforms.

A low-level, hardware-accelerated DirectX 12 library for machine learning. Windows ML uses DirectML as its primary GPU execution provider via the ONNX Runtime.

• Vendor-Neutral: Provides a single API to leverage GPU acceleration from AMD, Intel, and NVIDIA on Windows.
• Kernel Optimization: Implements highly tuned versions of neural network operators (like Conv2D, MatMul) for DirectX 12-compatible hardware.

A set of utilities, including the WinML Dashboard and the mlgen code generator, provided in the Windows Machine Learning component of the Windows SDK.

• Model Analysis: The dashboard profiles model performance, visualizes layers, and validates ONNX model compatibility.
• Code Generation: mlgen creates ready-to-use C#, C++, or WinRT API boilerplate code to load and run a specific model, accelerating integration.

Windows ML's architecture for automatically selecting the optimal available hardware (CPU, GPU, or a dedicated Neural Processing Unit (NPU)) for model inference.

• Performance Enumeration: The API can query available devices and their capabilities.
• Execution Device Selection: Developers can bind a model session to a specific device (e.g., LearningModelDeviceKind.DirectX) or let the system choose the default.