Quantization Scale

The quantization scale (S) is a multiplicative factor defined by the ratio of the original floating-point range (R) to the quantized integer range (N). For a tensor with a minimum value (min) and maximum value (max), the scale is typically calculated as:

• S = (max - min) / (2^b - 1) for unsigned b-bit integers.
• S = (max - min) / (2^b - 2) for signed b-bit integers (common for weights).

This formula ensures the full dynamic range of the integer type is utilized. The inverse operation, dequantization, reconstructs an approximate float value (r') from an integer (q) using: r' = S * q + Z, where Z is the zero-point.

The scale parameter dictates the fundamental trade-off between numerical resolution and representable range.

• Fine Resolution: A small scale value means each integer step corresponds to a tiny change in the float domain, allowing precise representation of values but limiting the maximum magnitude that can be captured.
• Wide Range: A large scale value allows the model to represent a broader spectrum of float values but with coarser granularity, increasing quantization error for small values.

Choosing the optimal scale involves analyzing the tensor's data distribution to cover its effective range without wasting precision on unused extremes.

The granularity at which the scale is applied is a critical design choice impacting accuracy and hardware efficiency.

• Per-Tensor Quantization: A single scale (and zero-point) is used for all values in an entire tensor. This is simple and highly efficient for hardware acceleration but can lead to higher error if the tensor's values have a wide or uneven distribution.
• Per-Channel Quantization: A unique scale is calculated for each output channel of a weight tensor (e.g., each filter in a convolutional layer). This accounts for variation across channels, significantly improving accuracy—especially for weights—at the cost of storing more scale parameters and slightly more complex arithmetic.

This defines whether the quantized range is centered on zero, impacting how the scale and zero-point are used.

• Symmetric Quantization: The float range is symmetric around zero (e.g., [-α, +α]). The scale S = α / (2^(b-1) - 1) and the zero-point Z is fixed at 0. This simplifies computation by eliminating the zero-point term in integer matrix multiplication.
• Asymmetric Quantization: The float range is not centered (e.g., [min, max]). It uses the scale S = (max - min) / (2^b - 1) and a non-zero zero-point Z to map the real zero accurately. This is more efficient for tensors like activations (ReLU outputs) that have a non-symmetric, non-negative distribution.

In Post-Training Quantization (PTQ), scales are not learned but are determined through a calibration process. A small, representative calibration dataset is passed through the model to observe the actual range of activations.

Common algorithms for determining the optimal scale (and zero-point) include:

• Min-Max: Uses the absolute minimum and maximum values observed. Simple but sensitive to outliers.
• Entropy / KL-Divergence: Adjusts the threshold to minimize the information loss between the original and quantized distributions, often yielding better accuracy.
• Percentile (e.g., 99.99%): Uses a percentile to clip outliers, providing a more robust range estimate.

The quantization scale is central to efficient integer-only inference. Modern AI accelerators (NPUs, GPUs with Tensor Cores) have dedicated hardware for integer matrix multiplication.

• The core computation for a layer becomes: INT8_Output = (INT8_Weights * INT8_Input) * (S_w * S_x / S_y).
• The scale fusion term (S_w * S_x / S_y) is often pre-computed and fused into the bias or a subsequent scaling operation, avoiding floating-point arithmetic in the critical path.
• This allows the entire forward pass to be executed using low-bit integer math, drastically reducing power consumption and latency compared to floating-point operations.

The ecosystem provides high-level APIs for quantizing LLMs. The bitsandbytes library is commonly used for 4-bit and 8-bit quantization.

• bitsandbytes.nn.Linear8bitLt: This module replaces standard linear layers. It computes per-block scales for weight quantization using vector-wise quantization, where a scale is stored for each contiguous block of values (e.g., 64 elements).
• load_in_8bit / load_in_4bit: Arguments in from_pretrained. Behind the scenes, this uses bitsandbytes to quantize weights on-load, calculating scales to normalize each block.
• NF4 Quantization: A non-uniform 4-bit type used in QLoRA. The scale is part of a more complex normalization process that maps weights to a predefined, optimized set of levels.

AI compilers lower quantized graphs to hardware-specific code, optimizing around the scale parameter.

• Apache TVM: Its relay.quantize pass inserts Q/DQ nodes. The scale is used during graph rewriting to fuse quantization patterns into efficient integer operators for backends like ARM DSP.
• XLA (TensorFlow/JAX): Performs quantization pattern matching in its HLO IR. Scales are folded into constant operations, enabling kernel fusion that pre-multiplies inputs by the inverse scale.
• IREE (MLIR): Leverages the MLIR quantization dialect (quant). Scales are represented as attributes in the IR, allowing for platform-agnostic quantization transformations before targeting Vulkan SPIR-V or CUDA.

In asymmetric quantization, the zero-point is the integer value that corresponds to the real numerical zero in the floating-point range. It aligns the quantized and floating-point number systems.

• Purpose: Allows the quantized range to precisely represent asymmetric data distributions (e.g., ReLU activations that are all non-negative).
• Calculation: Derived alongside the scale factor during calibration. The formula is typically zero_point = round(-min / scale).
• Impact: Using a zero-point adds a small computational overhead during integer operations but can significantly reduce quantization error for non-symmetric data.

A compression technique that reduces the precision of a pre-trained model's weights and activations without retraining. The quantization scale is determined using a small calibration dataset.

• Process: A representative dataset is run through the model to observe activation ranges. Scale and zero-point are calculated, then weights are quantized.
• Use Case: The fastest path to a smaller, faster model. Common targets are INT8 inference and FP16.
• Trade-off: Simplicity vs. potential accuracy loss, as the model cannot adapt to the quantization noise.

A model optimization technique that simulates quantization during training. Fake quantization nodes inject rounding and clipping noise, allowing weights to adapt.

• Mechanism: Uses a Straight-Through Estimator (STE) to approximate gradients through the non-differentiable quantization function.
• Advantage: Produces models that are robust to precision reduction, often matching or exceeding the accuracy of Post-Training Quantization.
• Cost: Requires retraining or fine-tuning, which adds computational overhead compared to PTQ.

The execution of a neural network using 8-bit integer precision for weights and activations. This is the primary goal of quantization techniques that employ a quantization scale.

• Performance Gain: Reduces model memory footprint by 4x (from FP32) and enables the use of high-throughput integer arithmetic units on modern CPUs, GPUs, and NPUs.
• Workflow: Weights and activations are quantized to INT8 using scale/zero-point. Integer operations are performed. Results are dequantized back to floating-point for non-linearities or output.
• Framework Support: Enabled by runtimes like TensorRT Quantization, TFLite Quantization, and ONNX Runtime.

A small, representative set of data used during Post-Training Quantization to determine the optimal quantization scale and zero-point for activation tensors.

• Function: Captures the dynamic range (min/max) or distribution (e.g., percentile) of activations in each layer.
• Size: Typically 100-1000 unlabeled samples. Must be representative of production data to avoid clipping error.
• Methods: Common calibration algorithms include Min-Max (use observed min/max) and Entropy (minimize KL divergence between float and quantized distributions).

The process of converting quantized integer values back into floating-point numbers. It is the inverse operation of quantization and uses the same scale and zero-point.

• Formula: float_value = scale * (int_value - zero_point).
• Where it Happens: Often required after integer matrix multiplications or convolutions, before applying non-linear activation functions (e.g., SiLU, LayerNorm).
• Optimization: In fused kernels, dequantization may be deferred or combined with subsequent floating-point operations to minimize memory traffic.