Activation Quantization

The primary goal of activation quantization is to enable integer-only arithmetic during inference. By converting both weights and activations to integers (e.g., INT8), the model eliminates the need for floating-point units (FPUs), which are often absent or power-inefficient on microcontrollers. This allows the entire inference pipeline to run using efficient integer multiply-accumulate (MAC) operations, drastically reducing compute latency and power consumption on edge hardware.

Quantizing activations from 32-bit floating-point (FP32) to 8-bit integers (INT8) reduces the data movement between memory and the processor by a factor of 4x. This is critical for TinyML deployments on microcontrollers, where:

• SRAM is limited (often < 512KB).
• Memory access dominates power consumption.
• Reducing bandwidth directly translates to lower latency and extended battery life for sensor-based applications.

Activation quantization is implemented via static or dynamic schemes, a key architectural decision.

• Static Quantization: Scaling factors for activations are predetermined using a calibration dataset and remain fixed during inference. This offers minimal runtime overhead and is standard for production TinyML.
• Dynamic Quantization: Scaling factors are computed on-the-fly for each input. While more flexible for varying input ranges, it introduces computational overhead often prohibitive for microcontrollers.

Determining the correct integer range for activations requires a calibration process. A representative dataset is passed through the FP32 model to observe the dynamic range (min/max values) of each layer's outputs. Statistical methods like min-max or percentile (e.g., 99.9th percentile) are used to set the scale and zero-point parameters, clipping outliers to minimize quantization error. Poor calibration is a primary source of accuracy degradation.

This defines how the floating-point range is mapped to the integer range.

• Asymmetric Quantization: Uses separate scale and zero-point values. It can precisely represent zero, which is crucial for layers like ReLU, but requires an extra integer addition per operation.
• Symmetric Quantization: Forces the zero-point to be 0. This simplifies the arithmetic (no offset addition) and is often used for weight quantization. For activations, it can be less accurate if the value range is not symmetric around zero.

Not all layers quantize equally. Certain operations are sensitive and require careful handling:

• Non-linearities (e.g., ReLU, Sigmoid): Often constrain activation ranges, making them easier to quantize.
• Residual Connections: Require careful alignment of quantization parameters where tensors from different branches are added.
• Attention Mechanisms in Tiny Language Models: The dynamic range of attention scores and softmax outputs can be large, posing a significant challenge for low-precision integer representation.

A training methodology that simulates quantization error during the forward and backward passes, allowing a model to learn robust representations for low-precision deployment. Unlike Post-Training Quantization (PTQ), QAT bakes the quantization process into the training loop.

• Key Mechanism: Uses fake quantization nodes that apply quantization and dequantization operations, but store high-precision weights for gradient updates.
• Primary Benefit: Typically achieves higher accuracy than PTQ for aggressive quantization schemes (e.g., INT8 or lower) by letting the model adapt.
• Trade-off: Requires a retraining phase, which adds computational cost and complexity to the development pipeline.

A compression technique where a pre-trained model is converted to a lower numerical precision after training is complete, using a small, representative calibration dataset.

• Process: A calibration run determines the dynamic range (min/max) of activations and weights to calculate optimal scaling factors and zero-points for linear quantization.
• Static vs. Dynamic: Static PTQ fixes these scaling factors for all inputs, while Dynamic PTQ recalculates them per input at runtime (higher overhead, better flexibility).
• Use Case: The standard, low-friction method for deploying models where a slight accuracy drop is acceptable, as it requires no retraining.

The process of converting a neural network's stored parameters (weights) from high-precision floating-point to lower-precision integers. It is often paired with activation quantization for fully quantized inference.

• Impact: Primarily reduces the model's storage footprint and the memory bandwidth required to load weights from storage/RAM to the compute unit.
• Asymmetric Benefit: Weights are static and their range is known post-training, making their quantization generally more straightforward and less impactful on accuracy than quantizing dynamic activations.
• Hardware Fit: Integer weights enable the use of highly efficient integer arithmetic logic units (ALUs) common in microcontrollers and edge AI accelerators.

An execution paradigm where the entire neural network forward pass is performed using integer arithmetic, eliminating floating-point operations. This requires both weight quantization and activation quantization.

• Core Requirement: The quantization scheme (scale, zero-point) must be designed so that the convolution/matrix multiplication of integer inputs and weights can be rescaled with a simple integer multiplication and bit-shift.
• Hardware Advantage: Enables deployment on microcontrollers and low-power chips that lack dedicated floating-point units (FPUs), drastically reducing power consumption and latency.
• Frameworks: Supported by TensorFlow Lite for Microcontrollers, PyTorch Mobile, and various proprietary neural processing unit (NPU) toolchains.

The two critical parameters in affine (linear) quantization that define the mapping between the floating-point and integer number systems.

• Scale (S): A floating-point number. Defines the ratio between a quantized integer unit and its floating-point value: float_value = scale * (int_value - zero_point).
• Zero-Point (Z): An integer. Represents the quantized value that corresponds to the real floating-point zero. This ensures that a true zero (e.g., from a ReLU activation) is exactly representable, preserving sparsity.
• Calculation: Determined during calibration (PTQ) or learned during training (QAT) by observing the min/max ranges of tensors: scale = (float_max - float_min) / (quant_max - quant_min).

Defines the granularity at which quantization parameters (scale, zero-point) are applied.

• Per-Tensor Quantization: A single scale and zero-point is used for an entire tensor (e.g., all weights in a layer, all activations from a layer). This is simpler but can be less accurate if the tensor's distribution varies significantly across channels.
• Per-Channel Quantization: Crucial for weights. A unique scale and zero-point is applied to each output channel of a weight tensor (or each channel of an activation tensor). This accounts for variation across channels, dramatically improving accuracy, especially for convolutional layers.
• Hardware Support: Per-channel weight quantization is widely supported; per-channel activation quantization is less common due to increased runtime complexity.