Quantization Grid

A quantization grid is the finite, predefined set of discrete levels used to approximate continuous-valued tensors (weights and activations) in a neural network. Its primary purpose is to enable integer-only inference by mapping high-precision 32-bit floating-point numbers to a limited set of low-bit integers (e.g., 8-bit, 4-bit, or binary values).

• Core Function: Replaces continuous distributions with a sparse, hardware-friendly representation.
• Key Benefit: Drastically reduces model memory footprint and enables the use of efficient integer arithmetic units (e.g., on NPUs), accelerating inference.

The granularity of a grid is defined by its step size (Δ or scale) and zero-point. These parameters determine the mapping between integer indices and real values.

• Step Size (Δ): The distance between adjacent grid points. A smaller Δ allows finer representation but may require more bits.
• Zero-Point: An integer value that corresponds exactly to the real value zero, enabling efficient padding and ReLU operations.
• Mathematical Mapping: A real value (r) is quantized to an integer (q) via: (q = \text{clamp}(\lfloor r/Δ \rceil + z, q_{\text{min}}, q_{\text{max}})).

For ternarization, the grid is simply ({-α, 0, +α}), where (α) is a learned scaling factor.

The spacing of values on the grid is a critical design choice, leading to two primary categories.

• Uniform Quantization: Grid levels are evenly spaced (e.g., {0, Δ, 2Δ, 3Δ, ...}). This is the most common method due to its simplicity and hardware efficiency, as dequantization is a simple multiplication.
• Non-Uniform Quantization: Grid levels are unevenly spaced, often concentrating levels where the parameter distribution is densest (e.g., near zero). Methods include logarithmic quantization (powers of two) and learned codebooks. This can better preserve information at ultra-low bit-widths (<4 bits) but requires more complex, lookup-based hardware.

Trade-off: Uniform grids favor speed; non-uniform grids can favor accuracy at extreme compression.

The scope over which a single grid is applied significantly impacts accuracy. This is known as quantization granularity.

• Per-Tensor Quantization: A single grid (one scale, one zero-point) is used for an entire weight tensor or activation layer. This is the simplest approach but can lead to high error if the tensor's value range is wide.
• Per-Channel Quantization: A unique grid is used for each output channel of a convolutional or linear layer. This is the standard for weight quantization, as it accounts for varying weight distributions across filters, greatly improving accuracy with minimal overhead.
• Per-Token/Per-Axis: For activations in transformers, scaling can be applied per-token (sequence element) to handle dynamic ranges in input data.

The process of defining the grid's range (min/max) and scale is called calibration. The method depends on the quantization workflow.

• Post-Training Quantization (PTQ): Uses a small set of unlabeled calibration data (e.g., 100-500 samples) to observe the dynamic range of activations. Common algorithms include:

  • Min-Max: Sets grid min/max to observed tensor min/max.
  • Entropy Minimization (e.g., TensorRT): Selects a range that minimizes the KL divergence between original and quantized distributions.
  • AdaRound: Optimizes the rounding of weights to grid points to minimize task loss.

• Quantization-Aware Training (QAT): The grid parameters (e.g., step size in LSQ) are made trainable. During fine-tuning, the model learns to adapt its weights and the optimal grid for quantization, yielding the most accurate low-bit models.

At very low bit-widths (1-4 bits), the grid's design becomes paramount and often involves specialized techniques.

• Binarization (1-bit): The grid is ({-α, +α}). Multiplication is replaced with bitwise XNOR and popcount operations.
• Ternarization (2-bit): The grid is ({-α, 0, +α}), introducing a sparsity-inducing zero value.
• Low-Bit Uniform (e.g., 4-bit): The grid has 16 evenly spaced levels. Channel-wise scaling is essential here to maintain accuracy.
• Challenges: The severe reduction in representational capacity amplifies the impact of quantization error. Techniques like PACT (learned activation clipping) and QAT are almost mandatory to achieve usable accuracy with such coarse grids.

Uniform quantization is a method where the discrete levels of the quantization grid are evenly spaced. This creates a simple linear mapping from continuous values to integers, defined by a scale (Δ) and zero-point. Its simplicity enables:

• Fast, deterministic quantization/dequantization operations.
• Efficient integer-only inference on hardware.
• However, it can introduce higher error if the original value distribution is not uniform, as all regions of the range are treated equally.

Non-uniform quantization allocates grid points unevenly, often concentrating them in regions where the parameter or activation distribution is densest. This adaptive approach aims to minimize quantization error for a given bit-width by better representing the underlying statistics. Common methods include:

• Logarithmic quantization, where levels are powers of two, enabling multiplications via bit-shifts.
• Clustering-based methods like K-means.
• While more accurate at very low bits, dequantization typically requires a lookup table, adding overhead.

Quantization-Aware Training (QAT) is the process of fine-tuning a model with simulated quantization operations in the forward pass. This allows the model's weights to adapt to the precision loss induced by the target grid before deployment. Key components include:

• Inserting fake quantization nodes that round values to the grid during forward passes but pass full-precision gradients during backpropagation.
• Using the Straight-Through Estimator (STE) to approximate gradients through the non-differentiable rounding function.
• Techniques like PACT (learned clipping) and LSQ (learned step size) to optimize the grid parameters themselves.

Post-Training Quantization (PTQ) applies a quantization grid to a pre-trained model without retraining. It uses a small calibration dataset to analyze activation distributions and determine optimal grid parameters (scale/zero-point). Advanced methods like AdaRound optimize weight rounding to minimize task loss. PTQ is faster than QAT but may incur greater accuracy loss, especially for extreme quantization below 8 bits, as the model cannot adapt to the grid constraints.

Integer-only inference is an execution paradigm where all computations of a quantized model—matrix multiplications, convolutions, and activations—are performed using integer arithmetic. This is enabled by a carefully designed quantization grid that maps floating-point ranges to integers. It eliminates the need for floating-point units (FPUs) on target hardware, leading to:

• Significant reductions in power consumption and latency.
• Deployment on microcontrollers and low-cost edge chips.
• The quantization grid's scale factors are absorbed into integer operations or handled with fixed-point arithmetic.

The scaling factor (often denoted α or Δ) is a critical parameter that defines the resolution of a uniform quantization grid. It determines the size of the step between adjacent discrete levels. The relationship is quantized_value = round(float_value / scale). Strategies include:

• Layer-wise scaling: One factor per layer.
• Channel-wise scaling: A unique factor per output channel, offering finer granularity and better accuracy for low-bit networks.
• The scale is learned during QAT (e.g., in LSQ) or calculated from statistics in PTQ to minimize the distortion between original and quantized values.