Model Quantization

Quantization reduces the numerical precision of a model's parameters (weights) and activations. The most common transition is from 32-bit floating-point (FP32) to lower-bit integer representations, such as 8-bit integers (INT8). This directly shrinks the model's memory footprint—a 4x reduction for INT8—and enables the use of faster, lower-power integer arithmetic units prevalent in edge hardware like microcontrollers and mobile CPUs. More aggressive quantization to 4-bit or binary (1-bit) values pushes compression further but requires sophisticated techniques to manage accuracy loss.

Two primary methodologies exist:

• Post-Training Quantization (PTQ): A faster, simpler process applied to a pre-trained model. It involves calibrating the model on a small representative dataset to determine optimal scaling factors for converting floats to integers. PTQ is ideal for rapid deployment but may incur a noticeable accuracy drop.
• Quantization-Aware Training (QAT): The model is trained or fine-tuned with simulated quantization operations in the forward pass. This allows the model to learn to compensate for the precision loss during training, typically yielding higher accuracy than PTQ but requiring a full training cycle and more computational resources upfront.

Quantized models unlock the performance of specialized hardware. Modern edge AI accelerators and NPUs (Neural Processing Units), such as the Arm Ethos-U55, are optimized for low-bit integer math. By matching the model's numerical format to the hardware's native operations, quantization achieves:

• Lower Latency: Faster integer operations reduce inference time.
• Higher Throughput: More inferences per second.
• Superior Energy Efficiency: Measured in Inferences Per Joule (IPJ), integer math consumes significantly less power than floating-point calculations, which is critical for battery-powered devices.

This distinction relates to when scaling factors are calculated:

• Static Quantization: Scaling factors (for weights and activations) are determined once during a calibration step and remain fixed during inference. This is the most common and efficient method for deployment, as it minimizes runtime overhead.
• Dynamic Quantization: Scaling factors for activations are computed on-the-fly for each input during inference. This is more flexible and can handle inputs with highly variable ranges but introduces computational overhead, making it less common for ultra-constrained edge devices.

This defines the granularity of the quantization parameters:

• Per-Tensor Quantization: A single set of scaling factors is applied to an entire tensor (e.g., all weights in a layer). This is simpler but can be less accurate if the tensor's values have a wide dynamic range.
• Per-Channel Quantization: Different scaling factors are applied to each channel (e.g., each output channel of a convolutional filter). This finer-grained approach typically preserves accuracy better, especially for weight tensors, and is widely supported by modern hardware and frameworks like TensorFlow Lite and ONNX Runtime.

Quantization is rarely used in isolation. It is often combined with other model compression techniques to achieve extreme efficiency:

• Pruning: Removing unimportant weights creates a sparse model, which is then quantized.
• Knowledge Distillation: A large teacher model trains a small, quantizable student model.
• Neural Architecture Search (NAS): Automated search for hardware-efficient architectures that are inherently quantization-friendly. Frameworks like MCUNet demonstrate this co-design, jointly optimizing the neural network architecture and the quantized inference engine for microcontrollers.

Model pruning is a compression technique that removes redundant or less important parameters (weights) or neurons from a neural network. This creates a smaller, sparser model with a reduced computational footprint.

• Structured Pruning: Removes entire channels, filters, or layers, resulting in a model that is directly executable on standard hardware.
• Unstructured Pruning: Sets individual weights to zero, creating an irregular sparse pattern that requires specialized libraries or hardware to exploit.
• Synergy with Quantization: Pruning and quantization are often applied sequentially; pruning reduces the number of parameters, and quantization reduces the bit-width of the remaining ones, leading to multiplicative size reductions.

Knowledge distillation is a compression and training technique where a compact student model is trained to replicate the outputs and internal representations of a larger, more accurate teacher model.

• Logits-based Distillation: The student is trained to match the teacher's softened output probabilities (logits), which contain dark knowledge about class relationships.
• Feature-based Distillation: The student is forced to mimic intermediate activations or attention maps from the teacher's layers.
• Edge Application: This enables high-accuracy, lightweight models suitable for microcontrollers by transferring learned generalization from a powerful but impractical cloud model.

Fixed-point arithmetic is a numerical representation scheme that uses integers to approximate real numbers by implicitly fixing the position of a decimal (radix) point. It is the computational foundation for quantized model inference.

• Representation: A number is stored as an integer, with a predefined scaling factor (e.g., Q7.8 format: 7 integer bits, 8 fractional bits).
• Hardware Efficiency: Microcontrollers without Floating-Point Units (FPUs) can perform integer operations much faster and with lower power consumption.
• Quantization Mapping: The process of quantization converts 32-bit floating-point weights and activations into fixed-point integers, enabling this efficient computation.

Hardware-Aware Neural Architecture Search (HW-NAS) is an automated process for designing neural network architectures that directly optimizes for target hardware performance constraints like latency, memory, and energy.

• Search Objectives: The search algorithm uses hardware performance predictors or direct on-device measurements to evaluate candidate models.
• TinyML Focus: It is essential for discovering models that fit within the severe KB-level memory budgets of microcontrollers.
• Frameworks: Techniques like Once-for-All Networks and MCUNet are prominent HW-NAS methodologies that co-design the model and the inference engine for ultra-low-power devices.

Inferences Per Joule (IPJ) is the primary energy-efficiency metric for TinyML, measuring the number of successful model inferences a device can perform per joule of energy consumed.

• System-Level Metric: IPJ accounts for the total energy cost of the sensor, compute, and communication subsystems during an inference cycle.
• Direct Impact: This metric directly translates to battery life for deployed edge devices. Quantization dramatically improves IPJ by reducing the computational energy per inference.
• Benchmarking: IPJ is a key figure used to compare different model architectures, quantization schemes, and microcontroller platforms.