Quantization

A one-shot compression technique applied after model training is complete. PTQ converts pre-trained 32-bit floating-point weights and activations directly to 8-bit integers using a small, unlabeled calibration dataset to determine optimal scaling factors.

• No retraining required: Fastest path to deployment
• Calibration dataset: Typically 100-1000 representative samples
• Accuracy trade-off: May degrade performance on models sensitive to weight perturbation
• Common formats: INT8, FP16, BF16

Frameworks like TensorRT and ONNX Runtime provide automated PTQ pipelines that complete in minutes.

A technique that simulates quantization noise during the forward pass of training while maintaining full-precision weights for the backward pass. This allows the model to learn parameters that are inherently robust to the precision loss introduced by subsequent integer conversion.

• Higher accuracy retention than PTQ for aggressive bit-widths
• Straight-through estimator (STE): Approximates gradients through non-differentiable quantization nodes
• Best for: INT4 and lower precision targets
• Training overhead: 20-50% slower than standard training

QAT is the preferred method when deploying vision transformers or large language models at 4-bit precision.

Two fundamental approaches to when scaling factors are computed:

Dynamic Quantization

• Weights quantized ahead of time; activations quantized on-the-fly during inference
• Lower latency reduction but simpler implementation
• Ideal for LSTM and Transformer models where activation ranges vary per input

Static Quantization

• Both weights and activations quantized offline using calibration data
• Enables full integer arithmetic on CPU and edge accelerators
• Requires representative calibration dataset

Static quantization achieves maximum throughput gains but demands more engineering effort.

Quantization scaling factors can be applied at different levels of granularity, directly impacting accuracy preservation:

Per-Tensor Quantization

• Single scale and zero-point for an entire weight tensor
• Simplest implementation, fastest execution
• Significant accuracy drop when weight distributions vary across channels

Per-Channel Quantization

• Independent scale and zero-point for each output channel
• Preserves accuracy by accommodating channel-wise variance
• Standard for convolutional layers in production deployments

Per-Group Quantization (used in GPTQ, AWQ)

• Divides channels into groups of 64-128 for finer granularity
• Critical for 4-bit LLM compression without catastrophic accuracy loss

A strategic choice determining which tensors undergo precision reduction:

Weight-Only Quantization

• Only model weights stored in low precision; activations remain FP16/BF16
• Reduces memory footprint and bandwidth but compute still in higher precision
• Dominant approach for LLM inference (GPTQ, AWQ, GGUF formats)
• Enables running 70B parameter models on consumer GPUs

Weight-and-Activation Quantization

• Both weights and activations reduced to INT8 or INT4
• Enables full integer matrix multiplication on CPU and NPU
• Required for maximum throughput on edge devices
• More challenging to tune due to activation range sensitivity

Memory-bound workloads benefit most from weight-only; compute-bound workloads require full quantization.

Specialized algorithms designed for the extreme compression of large language models to 4-bit and 3-bit precision:

GPTQ (Post-Training Quantization for GPTs)

• One-shot weight quantization using approximate second-order information
• Processes weights column-by-column, compensating for quantization error in remaining weights
• Requires ~100 samples for calibration; completes in hours on a single GPU

AWQ (Activation-Aware Weight Quantization)

• Identifies salient weight channels based on activation magnitudes
• Applies per-channel scaling to protect critical weights before quantization
• Outperforms GPTQ on instruction-tuned models

Both methods enable serving 13B+ parameter models on edge hardware with minimal perplexity degradation.