Weight Quantization

Quantization maps continuous 32-bit floating-point (FP32) weights to discrete integer representations like 8-bit integers (INT8) or even 4-bit integers (INT4). This is achieved through an affine transformation defined by a scale factor and a zero-point. The scale factor maps the floating-point range to the integer range, while the zero-point ensures that the floating-point value of zero is exactly representable. The core formula is q = round(r/s + z), where r is the real value, s is the scale, and z is the zero-point.

QAT simulates quantization noise during the forward pass of training, allowing the model to learn parameters that are robust to the precision loss. The forward pass uses fake quantization nodes that round and clamp values to mimic integer arithmetic, while the backward pass uses the Straight-Through Estimator (STE) to bypass the non-differentiable rounding operation. This results in significantly higher accuracy than PTQ for low-bit quantization, but requires full retraining access to the labeled dataset.

The scale factor and zero-point can be computed at different levels of granularity, representing a trade-off between accuracy and computational overhead:

• Per-Tensor: A single scale and zero-point for an entire weight tensor. Fastest but least accurate.
• Per-Channel: A separate scale and zero-point for each output channel in a convolutional layer. This is the standard for INT8 quantization of weights, as it respects the varying dynamic ranges across filters.
• Per-Group: Divides channels into groups, each with its own quantization parameters. Used for more aggressive INT4 quantization.

This distinction applies to the quantization of activations, not weights. Weight quantization is always static (pre-computed).

• Static Quantization: Activation ranges are pre-computed offline using a calibration dataset. This enables full integer arithmetic at inference, maximizing speed on integer-only hardware.
• Dynamic Quantization: Activation ranges are computed on-the-fly for each input. This avoids the need for a calibration dataset but introduces runtime overhead for range calculation, limiting speedups on some hardware.

The primary motivation for INT8 and INT4 quantization is to leverage specialized integer matrix-multiply instructions on modern hardware:

• NVIDIA Tensor Cores: Support INT8 and INT4 operations with significantly higher throughput than FP16.
• Intel DL Boost: Provides Vector Neural Network Instructions (VNNI) for INT8 convolution.
• ARM Neon: Offers SIMD instructions for INT8 dot-product operations on edge and mobile processors.
• Qualcomm Hexagon: A dedicated DSP with INT8 tensor accelerators for mobile inference.

A structural compression technique that removes redundant or low-magnitude weights from a neural network. Unlike quantization which reduces precision, pruning eliminates connections entirely.

• Unstructured pruning zeroes out individual weights, creating sparse matrices
• Structured pruning removes entire neurons, channels, or layers for hardware-friendly speedups
• Combined with quantization, can yield 10x or greater model size reduction
• Requires fine-tuning after pruning to recover accuracy

A training paradigm where a compact student model learns to mimic a larger teacher model. The student is trained on the teacher's soft output probabilities rather than hard labels.

• Captures dark knowledge — the relative probabilities of incorrect classes
• Student can be quantized after distillation for further compression
• Particularly effective for deploying large foundation models to edge devices
• Temperature parameter controls softness of teacher probability distribution

A communication efficiency technique critical for federated learning that reduces the bandwidth required to transmit model updates. Applies sparsification or quantization to gradients before transmission.

• Top-k sparsification: only the largest gradient values are transmitted
• Gradient quantization: reduces gradient precision to 8-bit or lower
• Reduces communication overhead by 100-1000x in distributed training
• Often combined with error feedback to maintain convergence quality

A quantization method applied after training is complete, converting a full-precision model to lower precision without retraining. Requires a small calibration dataset to determine optimal scaling factors.

• Dynamic quantization: weights quantized ahead of time, activations quantized at runtime
• Static quantization: both weights and activations quantized using calibration data
• Minimal data requirements compared to quantization-aware training
• May incur accuracy loss on sensitive models without corrective measures

A training methodology that simulates quantization effects during the forward pass while maintaining full-precision weights for gradient updates. Produces models robust to precision reduction.

• Inserts fake quantization nodes that model rounding and clipping behavior
• Straight-through estimator approximates gradients through non-differentiable quantization
• Yields higher accuracy than post-training quantization
• Standard approach for aggressive INT4 and INT8 deployment targets

Specialized hardware designed to execute quantized neural network operations with maximum efficiency. Modern NPUs contain dedicated silicon for INT8 and INT4 matrix multiplication.

• Exploits sparsity patterns created by combined pruning and quantization
• Delivers 10-50x energy efficiency over general-purpose CPUs for inference
• Common in edge devices: Apple Neural Engine, Google Edge TPU, Qualcomm Hexagon
• Requires compiler toolchains that map quantized graphs to hardware-specific instructions