NPU SDK

The model compiler is the central engine that translates a neural network from a standard framework format (like TensorFlow, PyTorch, or ONNX) into highly optimized instructions for the target NPU. This process involves critical graph optimizations such as operator fusion, layer tiling, and scheduling to maximize data reuse and minimize memory traffic. The compiler also performs quantization, converting models from 32-bit floating-point to lower precision formats (e.g., INT8, INT4) to drastically reduce model size and increase throughput, a mandatory step for most NPUs.

The runtime library is a lightweight software layer that manages model execution on the NPU at inference time. It handles memory allocation, scheduling tasks between the host CPU and the NPU, and data marshaling. The kernel library contains a set of highly hand-optimized, low-level functions for each neural network operator (e.g., convolution, pooling, activation) that are tailored to the NPU's microarchitecture. These pre-compiled kernels are invoked by the runtime to achieve peak hardware performance.

These tools provide visibility into the model's performance and behavior on the NPU. A profiler measures detailed metrics such as:

• Layer-by-layer latency and throughput
• NPU and system memory usage
• Utilization of compute units and memory bandwidth

Debugging tools help identify issues like compilation errors, runtime failures, or accuracy drops post-quantization. They are essential for iteratively optimizing model performance and verifying correct execution.

A functional simulator allows developers to execute and debug compiled NPU code on a host PC, providing a software model of the NPU's behavior. A cycle-accurate simulator or emulator goes further, modeling the hardware's timing and pipeline to give precise performance estimates (latency, power) before silicon is available. These tools are critical for early software development, algorithm validation, and performance tuning without physical hardware.

The kernel-mode driver is the software component that allows the host operating system to communicate with and manage the NPU hardware. It handles resource allocation, interrupt service routines, and power management. The user-mode driver or low-level API (e.g., OpenCL, vendor-specific APIs) provides a programming interface for the runtime to submit workloads (command buffers) to the NPU. This stack is responsible for the fundamental task scheduling and synchronization between the CPU and NPU.

To accelerate development, NPU SDKs typically include reference applications that demonstrate end-to-end pipelines for common use cases like image classification or object detection. A model zoo provides a collection of pre-trained, pre-optimized, and benchmarked models that are guaranteed to run efficiently on the vendor's NPU. These resources serve as both starting templates and performance baselines for developers.

An AI coprocessor is a dedicated hardware accelerator, such as a microNPU (Neural Processing Unit) or DSP block, integrated into a microcontroller or system-on-chip. Its sole purpose is to offload and dramatically accelerate neural network inference tasks from the main CPU core, enabling complex models to run within tight power and latency budgets.

• Key characteristic: Specialized for matrix multiplication and convolution operations.
• Example: The Arm Ethos-U55 is a microNPU designed to pair with Cortex-M CPUs.
• Interaction with SDK: The NPU SDK provides the compiler and runtime needed to target this specific hardware.

A micro-compiler is a specialized compiler within the TinyML toolchain that translates a high-level neural network model (e.g., from TensorFlow Lite) into highly optimized, low-level code for microcontroller execution. In the context of an NPU SDK, this compiler specifically targets the accelerator's instruction set and memory hierarchy.

• Primary function: Performs hardware-aware optimizations like layer fusion and memory scheduling.
• Output: Generates executable code or tightly packed bytecode for the NPU.
• Contrast with general compiler: It is narrowly focused on the graph operations of a neural network, not general-purpose C/C++ code.

Graph optimization is the process of transforming a neural network's computational graph to reduce its memory footprint and improve execution speed on constrained hardware. This is a core function of the compiler within an NPU SDK.

• Common techniques:
  • Constant Folding: Pre-computes operations on constant tensors.
  • Operator Fusion: Merges consecutive layers (e.g., Conv2D + BatchNorm + ReLU) into a single kernel to minimize intermediate tensor writes to memory.
  • Dead Code Elimination: Removes unused graph nodes.
• Impact: These optimizations are critical for fitting models into limited SRAM and reducing inference latency.

The tensor arena is a statically or dynamically allocated block of memory (typically SRAM) managed by the TinyML inference runtime. It is used as a shared scratchpad to store all intermediate activation tensors and temporary data during model execution.

• Purpose: Avoids costly heap allocations and fragmentation during inference.
• Sizing: A key task in deployment is determining the minimum arena size required for a given model, which the NPU SDK's profiling tools help calculate.
• NPU-specific: When an NPU is used, the arena may be split between the MCU's memory (for control and some data) and the NPU's tightly coupled memory (TCM) for high-speed tensor access.

An on-device SDK is a broader category of vendor-specific software development kits that provide libraries, APIs, and tools for developing applications with local, on-device machine learning inference. An NPU SDK is a specialized type of on-device SDK focused exclusively on the neural accelerator.

• Typical components: Inference runtime, hardware abstraction layer (HAL), power management APIs, and sample code.
• Scope: May support multiple inference backends (e.g., CPU via CMSIS-NN, NPU, DSP).
• Example: The STM32Cube.AI tool is an on-device SDK that can generate code for both the Cortex-M CPU and any available AI accelerator on STM32 MCUs.

The TinyML deployment workflow is the end-to-end process of taking a trained model to a functioning on-device application. The NPU SDK is a central tool in the middle stages of this pipeline.

• Key stages:
  1. Model Training & Export: Create a model in a framework like TensorFlow.
  2. Conversion & Quantization: Convert to a deployable format (e.g., TFLite) and apply INT8 quantization.
  3. NPU Compilation & Optimization: Use the NPU SDK to compile the model for the target accelerator.
  4. Firmware Integration: Link the SDK's generated code and runtime into the embedded application.
  5. Profiling & Validation: Use the SDK's tools to measure real-world latency, power, and accuracy on the hardware.
• Goal: To produce a reliable, resource-efficient binary for mass deployment.