Quantization Scale and Zero-Point

The scale factor is a positive floating-point number that defines the ratio between the quantized integer range and the original floating-point range. It acts as the resolution of the quantization grid.

• Formula: ( S = \frac{(FP_{max} - FP_{min})}{(Q_{max} - Q_{min})} )
• Purpose: Converts an integer value back to its approximate floating-point equivalent: ( FP \approx S \times (Q - Z) ).
• Example: If a tensor's values range from -2.5 to 2.5 and it's being quantized to the INT8 range [-128, 127], the scale is ( S = (2.5 - (-2.5)) / (127 - (-128)) = 5.0 / 255 \approx 0.0196 ). This means each integer step represents a change of ~0.0196 in the original floating-point space.

The zero-point is an integer value within the quantized range (e.g., within [-128, 127] for INT8) that corresponds exactly to the real value of zero in the floating-point domain.

• Purpose: Ensures that a real zero (a common value in activations like ReLU outputs) can be represented exactly, preserving the mathematical properties of zero for operations like padding. It aligns the two numerical domains.
• Formula: ( Z = Q_{min} - \text{round}(FP_{min} / S) ).
• Result: The dequantization formula becomes ( FP = S \times (Q - Z) ). When ( Q = Z ), ( FP = 0 ).

The choice of how to determine the floating-point range ([FP_{min}, FP_{max}]) defines the quantization scheme.

• Asymmetric Quantization: Uses the actual min/max of the tensor data. This minimizes quantization error but introduces a non-zero zero-point (Z), adding a small overhead to arithmetic operations.
• Symmetric Quantization: Forces ( FP_{min} = -FP_{max} ). This simplifies the math by setting the zero-point (Z) to 0, making operations faster. However, it is less efficient if the data distribution is not symmetric around zero, as it wastes part of the integer range.
• Trade-off: Asymmetric is more accurate for general activations; symmetric is faster and often used for weights.

Scale and zero-point can be applied with different granularities, affecting accuracy and hardware compatibility.

• Per-Tensor Quantization: A single scale and zero-point is calculated for an entire tensor. This is the simplest and most widely supported method.
• Per-Channel Quantization: A unique scale and zero-point is calculated for each channel (e.g., each output channel of a convolution kernel). This accounts for varying dynamic ranges across channels, significantly improving accuracy, especially for weight tensors. It is more computationally complex to implement.

Calibration is the process of estimating the optimal scale (S) and zero-point (Z) for a pre-trained model, typically using a small, representative dataset.

• Process: The calibration data is passed through the model to collect statistics (min/max values, histograms) for each tensor to be quantized.
• Methods:
  • Min-Max: Uses the absolute observed min/max values. Simple but sensitive to outliers.
  • Entropy / KL Divergence: Selects a range that minimizes the information loss between the original and quantized distributions. This is the standard method in frameworks like TensorRT for activation quantization.
• Output: The calibration process produces a set of constant S and Z parameters for the model, enabling static quantization.

The core benefit of defining scale and zero-point is enabling integer-only inference, where all computations use low-bit integer arithmetic, eliminating floating-point operations on the target device.

• Fused Operations: For a layer like a convolution, the integer computation becomes: ( Q_{out} = Z_{out} + \text{multiply_and_shift}( \sum (Q_{weight} - Z_{weight}) \times (Q_{input} - Z_{input}) , \frac{S_{weight} \times S_{input}}{S_{out}} ) ).
• The Re-quantization Step: The ratio ( (S_{weight} \times S_{input}) / S_{out} ) is a constant that is typically fused into the previous integer operation using a fixed-point multiply-and-shift or rounding operation.
• Result: This allows the entire network to run using only integer addition and multiplication, which is drastically faster and more energy-efficient on microcontrollers (MCUs) and digital signal processors (DSPs).

The core mathematical operation that defines the mapping between real and quantized values. For a real value ( r ), the quantized integer ( q ) is calculated as:

( q = \text{round}(r / S) + Z )

• ( S ) (Scale): A positive floating-point number representing the ratio between the quantized integer step size and the real-valued step size. It's calculated as ( S = (\text{max}(r) - \text{min}(r)) / (\text{max}(q) - \text{min}(q)) ).
• ( Z ) (Zero-Point): An integer within the quantized range (e.g., 0-255 for INT8) chosen so that ( r = 0 ) maps exactly to ( q = Z ). This ensures that a real zero (common in padding and ReLU activations) can be represented without error, preserving crucial mathematical properties like zero-padding equivalence.

The process of determining the optimal scale (S) and zero-point (Z) for each tensor (weights and activations) using a representative dataset. This is a critical step in Post-Training Quantization (PTQ).

• Weights: Scale is calculated directly from the min/max of the pre-trained weight values.
• Activations: Require a calibration dataset (e.g., 100-500 unlabeled samples) to observe the dynamic range of layer outputs. Common algorithms include:
  • Min-Max: Uses the absolute observed min/max.
  • Moving Average Min-Max: Tracks a running average to smooth outliers.
  • Entropy (KL Divergence): Selects a range that minimizes the information loss between the original and quantized distributions, often yielding higher accuracy.

Once calibrated, these parameters are baked into the model and remain static during inference.

How quantized models execute efficiently on CPUs and MCUs using integer math. The core operation of a quantized fully-connected or convolutional layer is reformulated to avoid floating-point calculations.

The operation for a layer becomes:

( q_3 = Z_3 + \text{multiply_and_shift}( \sum (q_1 - Z_1)(q_2 - Z_2), M ) )

• ( q_1, q_2 ): Input activation and weight integers.
• ( Z_1, Z_2 ): Their respective zero-points.
• ( M ): A pre-calculated fixed-point multiplier that fuses the scales ( S_1 ) and ( S_2 ) from the inputs and weights with the inverse of the output scale ( S_3 ).
• The multiply_and_shift is a high-performance integer operation that approximates multiplication by ( M ).

This allows the entire inference graph to run with INT8 or INT16 operations, drastically accelerating computation on integer-only hardware.

How specialized AI accelerators ingest and utilize quantization parameters for peak performance. These chips have dedicated integer arithmetic units (e.g., INT8, INT4 vector processors).

• Compilation: Frameworks like TensorFlow Lite for Microcontrollers (TFLM) or vendor-specific toolchains (e.g., NVIDIA TensorRT, Qualcomm AI Engine Direct) compile the quantized graph, baking scale and zero-point values into the instruction stream or lookup tables.
• Asymmetric vs. Symmetric: Many NPUs prefer symmetric quantization (where zero-point Z is forced to 0) for simpler, faster hardware. This requires the framework to perform a graph transformation, absorbing the zero-point operations into bias terms during compilation.
• Per-Channel Quantization: For weights in convolutional layers, hardware often supports a separate scale and zero-point per output channel, allowing for finer-grained adjustment and higher accuracy. This is a key optimization supported by TFLite and most modern NPUs.

The direct resource savings enabled by quantizing with scale and zero-point. This is the primary motivation for TinyML deployment.

• Model Size Reduction: Converting FP32 parameters (4 bytes each) to INT8 (1 byte each) yields a theoretical 4x reduction in model footprint. With per-tensor scale/zero-point overhead (typically 8 extra bytes per tensor), the net saving is ~3.5-4x.
• Memory Bandwidth: Loading 8-bit weights and activations from memory consumes one-fourth the bandwidth of 32-bit equivalents. This is often the dominant factor in inference latency and power consumption on microcontrollers, where memory access is energy-intensive.
• Cache Efficiency: Smaller models and tensors have a higher chance of fitting into faster, smaller on-chip caches (SRAM), reducing stalls and further accelerating inference.

These savings make deploying multi-million parameter models on devices with <1MB of SRAM feasible.

Quantization is a model compression technique that reduces the numerical precision of a neural network's weights and activations. It converts values from high-precision floating-point formats (like 32-bit FP32) to lower-precision integers (like 8-bit INT8). This process shrinks the model's memory footprint and accelerates inference by enabling integer arithmetic, which is faster and more power-efficient on most hardware. The core challenge is minimizing the quantization error—the difference between the original and quantized values—to preserve model accuracy.

Post-Training Quantization (PTQ) is a compression method where a pre-trained model is converted to a lower precision after training is complete. It uses a small, representative calibration dataset (with no labels required) to calculate the optimal quantization parameters—namely the scale and zero-point—for each tensor. PTQ is popular because it is fast and requires no retraining, but it can lead to higher accuracy loss compared to Quantization-Aware Training (QAT), especially for models with high dynamic ranges in their activations.

Quantization-Aware Training (QAT) is a technique where quantization error is simulated during the training or fine-tuning process. This is done by inserting fake quantization nodes into the model graph that mimic the rounding and clipping operations of integer quantization in the forward pass, while allowing gradients to flow through in the backward pass (using Straight-Through Estimator). This allows the model's weights to adapt to the lower precision, typically resulting in higher accuracy upon final integer conversion compared to PTQ, at the cost of additional training time.

These are two sub-categories of PTQ defined by when scale factors are calculated:

• Static Quantization: The scale and zero-point for both weights and activations are determined once during calibration and remain fixed for all inputs during inference. This is the most common and performant method for deployment.
• Dynamic Quantization: The scale and zero-point for activations are calculated on-the-fly for each individual input at runtime. This provides flexibility for inputs with highly variable ranges but introduces computational overhead. Weights are typically still statically quantized.

INT8 Inference refers to the execution of a neural network using 8-bit integer arithmetic for both weights and activations. It is the most common target for quantization due to its favorable balance:

• 4x model size reduction compared to FP32.
• 2-4x inference speedup on hardware with INT8 support (e.g., CPU VNNI instructions, GPU Tensor Cores, NPUs).
• Typically results in <1-2% accuracy loss for many models after PTQ or QAT. The scale and zero-point are essential for correctly transforming FP32 values into the INT8 range (typically -128 to 127) and back for operations like dequantization.

This distinction defines the quantization formula:

• Affine Quantization (Linear Quantization): Uses both a scale (floating-point) and an integer zero-point. The zero-point allows an exact mapping of a real value of zero (e.g., for padding in ReLU activations), which is critical for accuracy. This is the method described by 'quantization scale and zero-point.'
• Scale-Only Quantization (Symmetric Quantization): Uses only a scale factor, implicitly setting the zero-point to 0. This simplifies the math and hardware implementation but cannot precisely represent asymmetric value distributions, potentially leading to reduced quantization resolution and accuracy for certain tensors.