Post-Training Quantization (PTQ)

PTQ requires a small, representative calibration dataset (typically 128-512 samples) to analyze the statistical distribution of activations. This dataset is used to calculate scaling factors (quantization parameters) that map floating-point values to integer ranges with minimal information loss. No gradient-based learning occurs; the model's weights remain frozen during this profiling phase.

PTQ targets specific numerical formats to reduce memory and compute footprint. Common targets include:

• INT8 (8-bit integer): The most common target, offering a 4x memory reduction from FP32 with typically <1% accuracy drop for many models.
• INT4 (4-bit integer): Aggressive compression (8x reduction) requiring more sophisticated algorithms like GPTQ or AWQ to maintain accuracy.
• FP8 (8-bit floating point): An emerging standard that preserves a dynamic range similar to higher precision floats, beneficial for models with large activation outliers.

PTQ is categorized by when quantization parameters are determined:

• Static Quantization: Scaling factors are computed once during calibration and remain fixed during inference. This is the most common and performant form of PTQ, as it allows for kernel fusion and hardware acceleration.
• Dynamic Quantization: Scaling factors are computed on-the-fly for each input during inference. This handles variable input ranges better but introduces runtime overhead. It is often used for quantizing activations in models like LSTMs.

The scope of quantization defines the performance trade-off:

• Weight-Only Quantization: Only the model's weights are converted to low precision (e.g., INT8). Activations remain in floating-point (FP16/FP32). This reduces model size and memory bandwidth but offers limited compute speed-up.
• Full-Integer (Weight & Activation) Quantization: Both weights and activations are converted to integers (e.g., INT8). This enables the use of efficient integer arithmetic units (e.g., NVIDIA Tensor Cores, Intel VNNI) for maximal inference speed-up but is more sensitive to activation outliers.

Advanced PTQ algorithms mitigate accuracy loss:

• GPTQ: Uses layer-wise Hessian-based optimization to correct quantization errors, enabling accurate 4-bit weight quantization.
• AWQ: Identifies and preserves (does not quantize) salient weights—those multiplied by large activation magnitudes—through a scaling transformation.
• SmoothQuant: Statistically migrates the quantization difficulty from hard-to-quantize activations to easier-to-quantize weights via a per-channel smoothing factor, enabling performant 8-bit quantization of both.

Static quantization determines the quantization parameters (scale and zero-point) for both weights and activations by analyzing a single, representative calibration dataset. These parameters are then fixed for inference.

• Process: The calibration pass records the range of activations, after which the model is converted to use integer operations.
• Advantage: Eliminates runtime overhead for calculating quantization parameters, maximizing inference speed.
• Use Case: The standard method for quantizing convolutional networks (CNNs) and transformers where activation ranges are stable.

Dynamic quantization determines the quantization parameters for activations on-the-fly for each input during inference, while weights are quantized statically beforehand.

• Process: The scale and zero-point for a layer's output are computed based on the observed range of values for the current input batch.
• Advantage: Handles inputs with highly variable value ranges better than static quantization, often improving accuracy for certain layers (e.g., LSTM/GRU outputs).
• Trade-off: Introduces minor runtime overhead due to per-batch range calculation.

GPTQ is a layer-wise, approximate second-order quantization method designed for compressing large generative language models to very low precision (e.g., 4-bit).

• Mechanism: It uses the Hessian matrix (second-order derivatives) of the layer's weight reconstruction error to guide the quantization of weights in groups, minimizing the performance drop.
• Key Feature: Enables high compression (2-4 bits per weight) with minimal accuracy loss and is performed post-training without fine-tuning.
• Result: Produces models that run efficiently on consumer GPUs with libraries like bitsandbytes and auto-gptq.

AWQ is a PTQ method that identifies and protects a small subset of salient weights—those multiplied by large activation magnitudes—to preserve model quality at low bit-widths.

• Core Insight: Not all weights are equally important; the impact of a weight is scaled by its corresponding activation. Protecting 1% of salient weights can preserve most of the model's performance.
• Process: Scales weights and activations per channel to reduce the quantization error of these salient weights, enabling robust 4-bit quantization.
• Benefit: Like GPTQ, it requires no retraining and maintains strong zero-shot task performance for language models.

SmoothQuant is a PTQ technique that addresses the challenge of quantizing transformer models with large, outlier values in their activations, which are difficult to represent in low-precision integers.

• Problem: Outliers in activations (common in models like OPT and BLOOM) force high quantization error if activations are quantized directly.
• Solution: Mathematically migrates the quantization difficulty from activations to weights by 'smoothing' the activation scales via a per-channel scaling factor absorbed into the preceding layer's weights.
• Outcome: Enables 8-bit quantization of both weights and activations (W8A8) for full transformer inference, which is highly efficient on modern integer hardware.

The calibration dataset is a small, representative set of unlabeled data (typically 128-512 samples) used by PTQ algorithms to determine optimal quantization parameters.

• Purpose: Used to observe the statistical range (min/max) of activations for static quantization or to compute Hessian information for methods like GPTQ.
• Key Metrics: PTQ success is measured by:
  • Task Accuracy Drop: The change in performance (e.g., perplexity, accuracy) on a benchmark after quantization. A drop of <1% is often considered successful.
  • Model Size Reduction: e.g., reducing a 16-bit (FP16) model to 8-bit (INT8) cuts the model size in half.
  • Inference Latency/Speedup: The reduction in compute time achieved by using integer arithmetic on supporting hardware (e.g., CPUs, NPUs).

Quantization-aware training is a process where a neural network is trained or fine-tuned with simulated quantization operations in the forward pass. This allows the model to learn parameters robust to the precision loss incurred during subsequent integer deployment. Unlike PTQ, which is applied after training, QAT bakes quantization error into the training loop, often yielding higher accuracy at ultra-low precisions (e.g., INT4).

• Key Mechanism: Uses fake quantization nodes during training to model the rounding and clipping effects of integer arithmetic.
• Trade-off: Requires a full training cycle, making it more computationally expensive than PTQ but more accurate.
• Typical Use Case: Deploying models to ultra-constrained edge devices where every bit of precision matters.

GPTQ is a state-of-the-art post-training quantization method specifically designed for compressing large transformer-based language models. It uses layer-wise compression based on approximate second-order information (Hessian matrices) to correct the error introduced by quantizing each weight.

• Key Mechanism: Applies Optimal Brain Quantization principles, quantizing weights in a sequence while updating remaining unquantized weights to compensate for the loss.
• Precision: Enables reliable 4-bit and lower precision quantization of model weights with minimal performance degradation.
• Distinction: A leading algorithm for weight-only quantization, often used in conjunction with FP16 or 8-bit activations for inference.

AWQ is a post-training quantization method that scales model weights based on activation magnitudes to preserve critical information. It identifies that not all weights are equally important—salient weights (those multiplied by large activation outliers) are more sensitive to quantization.

• Key Mechanism: Applies a per-channel scaling to protect these salient weights before quantization, then inversely scales the following layer's activations.
• Advantage: A zero-shot method requiring no calibration data or backpropagation, making it fast and robust.
• Outcome: Enables high-performance 4-bit quantization of both weights and activations, crucial for on-device LLM deployment.

SmoothQuant is a post-training quantization technique that addresses the challenge of quantizing transformer activations, which often contain extreme outliers. It migrates the quantization difficulty from activations to the more stable weights.

• Key Mechanism: Applies a mathematical smoothing by scaling down the activations and scaling up the corresponding weights per channel, equalizing their dynamic ranges.
• Primary Benefit: Enables per-tensor or 8-bit quantization of both weights and activations, simplifying hardware deployment.
• Hardware Impact: Allows for efficient execution on integer-only hardware (e.g., many NPUs and GPUs) that cannot natively handle FP16 activations.

Pruning is a model compression technique that removes redundant or less important parameters (weights, neurons, or layers) from a neural network. It is often used in conjunction with quantization to maximize compression.

• Types: Includes unstructured pruning (individual weights) and structured pruning (entire channels or layers), with the latter being more hardware-friendly.
• Process: Can be applied post-training or during training (pruning-aware training).
• Synergy with PTQ: Creates a sparse model with fewer non-zero values, which is then quantized, leading to multiplicative size and latency reductions.

Knowledge distillation is a compression and transfer learning technique where a small, efficient student model is trained to mimic the behavior of a larger, more accurate teacher model. The student learns from both the teacher's output probabilities (soft labels) and the ground truth.

• Relation to PTQ: Provides an alternative path to a small, fast model. A distilled model can then be further compressed via PTQ for edge deployment.
• Key Benefit: Can capture the teacher's generalization ability and dark knowledge in a more compact architecture.
• Common Use: Creating tiny language models for on-device use, which are prime candidates for subsequent quantization.