Quantization

Quantization fundamentally reduces the numerical precision of a model's parameters (weights) and activations. The most common transition is from 32-bit floating-point (FP32) to 8-bit integers (INT8), which provides a 4x reduction in model size and a 4x reduction in memory bandwidth. Lower precisions like INT4 or binary (1-bit) are possible for extreme compression, but with greater accuracy trade-offs. This precision shift enables models to fit into the limited flash memory of microcontrollers and accelerates computation by leveraging efficient integer arithmetic units.

Quantized models unlock the performance of specialized hardware. Microcontrollers and neural processing units (NPUs) often have dedicated integer arithmetic logic units (ALUs) that execute INT8 operations much faster and more efficiently than floating-point ones. Frameworks like CMSIS-NN provide hand-optimized kernels for these operations. Key hardware benefits include:

• Lower power consumption per operation.
• Higher operations per second (OPS) within the same power budget.
• Direct compatibility with digital signal processor (DSP) instructions and SIMD extensions for parallel computation.

A critical step in quantization is determining how to map the continuous range of float values to a discrete set of integers. This is done via calibration using a representative dataset. The process defines two key parameters:

• Scale (S): A floating-point factor that maps integer values back to their approximate float range.
• Zero-Point (Z): An integer bias that ensures the real value zero is exactly representable, crucial for layers like ReLU. The formula for affine (asymmetric) quantization is: float_value ≈ S * (int_value - Z). Proper calibration minimizes the information loss from this mapping.

Quantization schemes define how the integer range is aligned with the float range.

• Symmetric Quantization: The range is symmetric around zero (e.g., [-127, 127] for INT8). The zero-point is fixed at 0. This simplifies the computation (no zero-point subtraction needed) and is commonly used for weight quantization.
• Asymmetric Quantization: The range is defined by actual min/max values (e.g., mapping min_float to 0 and max_float to 255). It uses a non-zero zero-point. This is more accurate for activation quantization because it can precisely capture the asymmetric output of ReLU activations (all non-negative).

Quantization introduces a quantization error—the difference between the original float value and its dequantized equivalent. This error propagates through the network, potentially degrading task accuracy (e.g., classification score). The trade-off is managed by:

• Choosing bit-width: INT8 typically has minimal loss (<1% for many models), while INT4 can be significant.
• Quantization Granularity: Per-tensor scaling is simple; per-channel scaling for weights is more granular and often preserves accuracy better.
• Quantization-Aware Training (QAT): Simulating quantization during training allows the model to adapt and recover accuracy, often outperforming Post-Training Quantization (PTQ).

A key goal for microcontroller deployment is integer-only inference, where the entire forward pass uses integer arithmetic, avoiding costly float operations. This requires:

• Fusing operations like convolution, batch normalization, and activation into a single integer kernel.
• Pre-computing weight and bias parameters as integers.
• Using fixed-point arithmetic for any necessary non-integer scaling, often implemented with integer multiplication and bit-shifting. This characteristic is essential for deterministic, low-latency execution on cores without a floating-point unit (FPU).

Most modern microcontrollers (e.g., Arm Cortex-M4, M7, M33, M55, RISC-V cores with P extension) include hardware that accelerates quantized inference:

• Integer Arithmetic Logic Units (ALUs): Native support for 8-bit and 16-bit integer operations, which are fundamentally faster and more energy-efficient than software-emulated floating-point.
• Single Instruction, Multiple Data (SIMD): Instructions like Arm's MVE (Helium) or DSP extensions (e.g., SMLAD, SMLAL) allow a single instruction to perform multiple multiply-accumulate (MAC) operations in parallel on integer data, crucial for convolution and matrix multiplication.
• Digital Signal Processors (DSPs): Often integrated as a co-processor (e.g., the Cadence Tensilica HiFi DSPs), these are highly optimized for the vector math of quantized neural networks, offering significant performance per mW advantages over the main CPU.

Dedicated neural processing units (NPUs) are becoming common in high-end microcontrollers and system-on-chips (SoCs) for the edge. Examples include the Arm Ethos-U55/U65, Synaptics Astra, and GreenWaves GAP9. These accelerators are designed from the ground up for quantized tensor operations:

• Systolic Arrays or Tensor Cores: Hardware that performs massive parallel matrix multiplications for INT8/INT4 data.
• Weight Activation Sparsity Support: Hardware that can skip computations involving zero values, exploiting the sparsity often induced by pruning.
• On-Chip Memory Hierarchies: Dedicated SRAM for weights and activations to minimize power-hungry external memory accesses. Using these accelerators requires framework support via delegates (TFLite) or BYOC mechanisms (TVM), which partition the model graph and offload quantized layers to the NPU.

A specific form of quantization where model parameters and activations are constrained exclusively to integer values (e.g., INT8, INT4). This enables execution using the integer-only arithmetic logic units (ALUs) ubiquitous in microcontrollers, completely avoiding the need for floating-point hardware. Key aspects:

• Enables massive speedups on standard MCU cores.
• Requires careful management of scaling factors to maintain accuracy.
• The target format for most production TinyML deployments.

A conversion process where a pre-trained, full-precision (FP32) model is statically quantized to a lower precision (e.g., INT8) using a small, representative calibration dataset. No retraining is involved, making it fast and straightforward. Typical workflow:

1. Run calibration data through the FP32 model to collect activation ranges (min/max).
2. Calculate scaling factors and zero-points for each tensor.
3. Convert weights and define quantized operations.

• Best for: Rapid deployment where a minor accuracy drop is acceptable.

A more advanced technique where quantization is simulated during the training or fine-tuning phase. The model learns to compensate for the precision loss, typically yielding higher accuracy than PTQ. Core mechanism:

• Fake quantization nodes are inserted into the training graph.
• These nodes quantize and immediately dequantize tensors during forward passes, mimicking the error.
• The backward pass uses the straight-through estimator (STE) to approximate gradients.
• Result: A model whose weights are already robust to low-precision representation.

The statistical process of determining the optimal quantization parameters (scale and zero-point) for each activation tensor in a model. It is the critical step in Post-Training Quantization. Methods include:

• Min/Max: Uses the absolute minimum and maximum values observed.
• Moving Average Min/Max: Tracks a running average to mitigate outliers.
• Entropy / KL Divergence: Minimizes the information loss between the FP32 and quantized distributions, often yielding the best accuracy.
• Poor calibration is a primary source of quantization accuracy loss.

Two fundamental schemes for defining the mapping between floating-point and integer ranges.

Symmetric Quantization:

• The quantization range is symmetric around zero (e.g., [-127, 127] for INT8).
• Uses a single scale factor. The zero-point is fixed at 0.
• Simpler, faster math. Commonly used for weight quantization.

Asymmetric Quantization:

• The range is defined by separate min and max values (e.g., [min, max]).
• Uses a scale and a non-zero zero-point.
• More accurately represents skewed data (e.g., ReLU activations, which are all >=0).

A numerical representation system where numbers are expressed as integers with an implicit, fixed binary point (i.e., a fixed number of fractional bits). It is the computational foundation of integer quantization on hardware without floating-point units (FPUs). How it works:

• A 32-bit integer might be interpreted as having 16 integer bits and 16 fractional bits (Q16.16 format).
• Multiplication of two Qm.n format numbers yields a result in Q(2m).(2n) format, often requiring a re-scaling (shifting) operation.
• Direct Link to Quantization: The scaling factors in quantization effectively create a dynamic fixed-point representation per tensor, managed by the software framework.