Quantization for Neural Processing Units (NPUs)

NPU-targeted quantization schemes are designed to map directly to the hardware's native integer compute units. Unlike CPUs or GPUs that may emulate integer operations, NPUs contain dedicated integer multiply-accumulate (MAC) arrays. The quantization parameters (scale, zero-point) and bit-width (e.g., INT8, INT4) are chosen to ensure that all tensor operations—convolution, matrix multiplication, pooling—execute entirely within these integer units without costly dequantization/requantization cycles during the core compute pipeline.

The granularity of quantization (per-tensor, per-channel) is dictated by the NPU's memory hierarchy and data loading patterns. For example:

• Per-channel quantization of weights is often mandatory to align with systolic array inputs that process channels in parallel.
• Activation quantization may be per-tensor if the hardware uses a unified buffer for layer outputs. The scheme avoids granularities that would require runtime scaling calculations not supported in hardware, ensuring deterministic latency.

NPUs have fixed-precision datapaths. Effective quantization aligns the bit-width of activations and weights to match this fixed width (e.g., INT8 for both). Mismatched precisions (e.g., INT4 weights with INT8 activations) require on-the-fly promotion in the data stream, causing stalls. Advanced NPUs may support mixed-precision modes (e.g., 4-bit weights, 8-bit activations), but the quantization scheme must explicitly target these predefined, hardware-validated mode combinations.

The NPU's proprietary compiler (e.g., NVIDIA TensorRT, Qualcomm AI Engine Direct, Google Edge TPU Compiler) is a first-class citizen in the quantization flow. The compiler:

• Determines optimal clipping ranges for activations based on fused operator graphs.
• Performs quantization folding, merging scale/bias operations into preceding quantize/dequantize nodes.
• Encodes the final scale and zero-point parameters directly into the compiled model binary, making them immutable runtime constants for the hardware.

NPU integer units use saturation arithmetic, where results exceeding the bit-width range are clamped to the maximum/minimum representable value. Quantization schemes must statistically guarantee that intermediate accumulator values (e.g., the 32-bit sum of many INT8 products) do not overflow. This involves analyzing weight/activation distributions during calibration to set conservative clipping ranges, a process often called range setting or calibration for saturation.

Many modern NPUs (e.g., Google TPU, Apple Neural Engine) include hardware support for weight sparsity (e.g., 2:4 structured sparsity). Quantization for these NPUs is often combined with pruning. The quantization process must preserve the sparsity pattern; zero values are quantized to the true integer zero (requiring asymmetric quantization if the original range isn't symmetric). This allows the hardware to skip computations entirely, compounding the benefits of compression and quantization.

Integer quantization constrains a neural network's weights and activations to integer values (e.g., INT8, INT4). This is the foundational technique that enables NPU acceleration, as NPUs contain specialized integer arithmetic logic units (ALUs) that execute these operations with high throughput and low power consumption, unlike general-purpose CPUs which are optimized for floating-point math.

Hardware-aware compression refers to model optimization techniques co-designed with the characteristics of the target silicon. For NPUs, this goes beyond simple bit-width reduction to consider:

• Dataflow architecture of the NPU (weight stationary, output stationary).
• Supported precision modes (e.g., INT8, INT4, mixed-precision).
• Memory hierarchy and bandwidth constraints. The goal is to structure the quantized model graph to maximize data reuse and minimize costly off-chip memory accesses.

Compute graph optimization transforms a model's computational graph for optimal execution on target hardware. For NPU quantization, this involves:

• Operator fusion: Merging consecutive operations (e.g., Conv + BatchNorm + Activation) into a single, quantized kernel to reduce intermediate memory writes.
• Quantization folding: Absorbing linear operations like batch normalization into the preceding layer's weights before quantization.
• Layout transposition: Rearranging tensor data formats (e.g., NHWC to NCHW) to match the NPU's native memory access patterns for peak bandwidth.

Calibration is the process of analyzing a representative dataset to determine the optimal quantization parameters (scale and zero-point) for each tensor. For NPU deployment, calibration is critical because:

• It defines the mapping from float to integer ranges.
• Poor calibration leads to clipping (saturation) or excessive quantization error.
• NPU-specific calibration may use per-channel quantization for weights to better match the data distribution in convolutional filters, improving accuracy over simpler per-tensor methods.

Sparse model inference executes neural networks where a significant portion of weights or activations are zero. NPUs often include dedicated hardware to skip computations involving zeros, providing a synergistic optimization with quantization:

• Weight pruning creates sparsity, which is then quantized.
• The NPU exploits both the reduced precision (fast integer math) and the sparsity (skipped operations).
• This combination can lead to multiplicative reductions in latency and energy consumption compared to dense, floating-point models.

On-device model formats are the serialized file formats and runtime representations for deploying quantized models to edge devices with NPUs. These formats encapsulate the quantized graph, parameters, and metadata required for the NPU driver. Common examples include:

• TensorFlow Lite (.tflite) with quantization.
• ONNX Runtime with QNN (Quantized Neural Network) execution provider.
• Vendor-specific formats (e.g., Qualcomm's .dlc, MediaTek's .tnn). These formats ensure the quantized model is correctly interpreted and executed by the device's NPU runtime engine.