Non-Uniform Quantization

Unlike uniform quantization, which spaces levels evenly, non-uniform quantization places more quantization points in regions of high parameter or activation density. This is mathematically analogous to minimizing the mean squared quantization error for a given bit budget by aligning the quantization grid with the probability density function of the values. For example, if weight values cluster near zero (a common Laplacian or Gaussian distribution), levels are concentrated there, preserving fine-grained differences that would be lost with uniform spacing.

A prevalent and hardware-friendly form of non-uniform quantization uses logarithmic spacing or restricts levels to powers of two. This transforms multiplications into efficient bit-shift operations during inference.

• Mechanism: Values are quantized to the nearest (2^n) or ( \text{sign}(x) \times 2^{\lfloor \log_2 |x| \rfloor} ).
• Benefit: Eliminates the need for hardware multipliers, drastically reducing circuit area and power consumption in integer-only inference engines.
• Trade-off: The coarse spacing for larger magnitudes can increase error for values far from zero, making it suitable for distributions where large values are rare or less critical.

Advanced non-uniform quantization methods treat the quantization grid itself as a learnable parameter during Quantization-Aware Training (QAT). Techniques like Learned Step Size Quantization (LSQ) or Parameterized Clipping Activation (PACT) optimize the step sizes or clipping thresholds by gradient descent.

• Process: The model jointly optimizes network weights and the parameters defining the non-uniform quantization levels.
• Outcome: This data-driven approach discovers an optimal, task-specific quantization grid that minimizes final task loss, often outperforming heuristic distributions like logarithmic.

The primary advantage of non-uniform quantization becomes most apparent at very low bit-widths (e.g., 2-4 bits) and in extreme quantization scenarios like ternarization. When only a handful of discrete values are available, allocating them intelligently based on the value distribution preserves significantly more information than a naive uniform spread. This makes it a cornerstone technique for pushing models into the sub-8-bit regime while maintaining usable accuracy, a critical requirement for TinyML deployment and edge AI.

The information-theoretic benefits of non-uniform quantization come with practical engineering costs.

• Dequantization Overhead: Mapping stored integer indices back to their non-uniform floating-point values requires look-up tables (LUTs) or more complex scaling logic, adding latency and memory access overhead.
• Non-Standard Kernels: Efficient execution requires custom compute graph optimization and kernel implementations, as standard deep learning frameworks and Neural Processing Units (NPUs) are often optimized for uniform quantization.
• Calibration Complexity: Post-Training Quantization (PTQ) calibration algorithms (e.g., to fit a logarithmic distribution) are more complex than simple min-max range estimation.

Understanding non-uniform quantization is best done by contrasting it with its uniform counterpart.

Logarithmic quantization maps values to powers of two (or a learned base). This creates a non-uniform grid where spacing increases exponentially, providing high resolution for small values and coarse resolution for large ones.

• Core Mechanism: Replaces floating-point multiplications with efficient bit-shift operations during inference, as multiplying by a power of two is equivalent to shifting bits.
• Hardware Advantage: Eliminates the need for hardware multipliers, making it ideal for deployment on ultra-low-power microcontrollers and custom digital signal processors (DSPs).
• Typical Use: Often applied to weights in convolutional and fully-connected layers where the distribution is heavy-tailed.

This technique uses clustering algorithms, like k-means, to determine optimal quantization levels. The centroids of the clusters become the discrete quantization values.

• Process: The full-precision values are grouped into k clusters (where k = 2^b for a b-bit representation). The centroid of each cluster is calculated and used as the quantized value for all members.
• Adaptive Grid: The resulting grid is non-uniform and directly adapts to the empirical distribution of the tensor (weights or activations), minimizing mean squared quantization error.
• Application: Commonly used in Post-Training Quantization (PTQ) where a small calibration dataset is used to cluster activation values and set per-layer or per-channel codebooks.

Vector Quantization (VQ) extends clustering to groups of parameters. Instead of quantizing individual scalars, it quantizes blocks or vectors of weights together into codewords from a shared codebook.

• Granularity: Operates on a higher level of abstraction (e.g., 4x4 or 8x8 blocks of weights), capturing correlations within the block.
• Compression Ratio: Can achieve very high compression rates, as a single codebook index represents an entire vector. This is a form of product quantization.
• Trade-off: While highly efficient for storage, it introduces complexity during inference as codewords must be de-referenced from the codebook. It's often used for compressing large embedding tables or fully-connected layers.

A strict subset of logarithmic quantization, Power-of-Two (PoT) quantization constrains quantized values to exact powers of two, including zero. The grid is defined as {0, ±2^k} for integer k.

• Deterministic Grid: Unlike learned logarithmic bases, PoT uses a fixed, hardware-friendly grid.
• Inference Speed: Enables the replacement of all multiplications with bit-shifts, offering deterministic latency and minimal power consumption. Additions become the primary operation.
• Limitation: The rigid grid can lead to higher quantization error compared to adaptive logarithmic methods, making it suitable for layers less sensitive to precision.

Additive Powers-of-Two (APoT) quantization expands the representational capacity of PoT by allowing quantized values to be the sum of a limited number of PoT terms.

• Representation: A value is represented as v = Σ a_i * 2^{b_i}, where a_i ∈ {0,1} and b_i is an integer. For example, with two terms, you can represent 3 = 2^1 + 2^0.
• Denser Grid: This creates a non-uniform grid that is denser around zero than pure PoT, offering a better accuracy-efficiency trade-off.
• Computation: While more expressive, inference requires a few additions of shifted values, which is still more efficient than full-precision multiplication.

This family of techniques treats the quantization grid itself as a set of learnable parameters during Quantization-Aware Training (QAT). The model jointly optimizes network weights and the spacing/values of quantization levels.

• Key Method: Learned Step Size Quantization (LSQ) and its extension LSQ+ learn a per-tensor or per-channel step size (Δ). For non-uniform grids, the actual quantization levels can be learned directly.
• Optimal Adaptation: By learning the grid, the technique automatically discovers and allocates more levels to critical regions of the value distribution, minimizing task loss.
• Result: Typically achieves the highest accuracy for a given target bit-width but requires end-to-end training or fine-tuning, unlike calibration-based PTQ methods.

Uniform quantization is a method where the quantization levels are evenly spaced across the range of values. This creates a simple linear mapping from full-precision values to integers, defined by a scale and zero-point. Its primary advantage is computational simplicity, as quantization and dequantization are straightforward arithmetic operations. However, for parameter distributions that are not uniform (e.g., Gaussian or Laplacian), this even spacing can lead to higher quantization error, as many levels are wasted on regions of the distribution with few values.

Logarithmic quantization is a specific type of non-uniform quantization where values are mapped to discrete levels that are powers of two. This creates a quantization grid with exponentially spaced levels (e.g., ..., 1/8, 1/4, 1/2, 1, 2, 4, ...). The key hardware advantage is that multiplications between quantized values and activations can be implemented as efficient bit-shift operations, eliminating the need for hardware multipliers. It is particularly well-suited for compressing weights, which often follow a log-normal distribution.

Quantization-Aware Training (QAT) is the process of fine-tuning or training a neural network with simulated quantization operations in the forward pass. During training, weights and activations are quantized and dequantized, but the backward pass uses the Straight-Through Estimator (STE) to propagate gradients through these non-differentiable operations. This allows the model's parameters to adapt to the precision loss, significantly recovering accuracy compared to applying quantization after training. Techniques like PACT (Parameterized Clipping Activation) and LSQ (Learned Step Size Quantization) are advanced QAT methods.

The Straight-Through Estimator (STE) is a critical gradient approximation technique used to train neural networks with discrete-valued parameters, such as in binarization or ternarization. During the backward pass of training, the non-differentiable quantization function is treated as an identity function, allowing gradients to pass through unchanged. While this provides a biased gradient estimate, it is empirically effective. It is often combined with techniques like gradient clipping to stabilize training. The STE enables end-to-end training of models that would otherwise be untrainable with standard backpropagation.

Integer-only inference is an execution paradigm where all computations of a quantized neural network are performed using integer arithmetic. This eliminates the need for floating-point units (FPUs) on the target hardware, which are power-hungry and area-intensive. The pipeline involves:

• Quantizing weights and activation ranges to integers.
• Performing convolutions and matrix multiplications with integer operations.
• Using integer-friendly approximations for non-linearities like ReLU.
• This is the ultimate goal of extreme quantization, enabling efficient deployment on microcontrollers, mobile SoCs, and dedicated NPUs.

In extreme quantization methods like binarization and ternarization, a scaling factor (often denoted α) is a critical learned or calculated multiplier. Since constraining values to {-1, +1} or {-1, 0, +1} drastically reduces dynamic range, a single real-valued scaling factor is applied per layer or per channel to recover magnitude. For example, in a Ternary Weight Network (TWN), the forward pass computes W ≈ α * T, where T contains ternary values. This factor minimizes the L2 error between the full-precision and quantized weights and is essential for maintaining model accuracy.