Model Compression

Pruning is the systematic removal of redundant or non-critical parameters (neurons, channels, or entire layers) from a neural network. The goal is to create a sparser, more efficient model without significantly degrading accuracy.

• Unstructured Pruning: Removes individual weights based on a criterion like magnitude (e.g., smallest absolute values). This creates an irregular, sparse model that often requires specialized sparse libraries for efficient inference.
• Structured Pruning: Removes entire structural components, such as neurons, filters, or attention heads. This results in a smaller, dense model that can run efficiently on standard hardware without specialized kernels.
• Iterative Pruning: A common methodology where the model is repeatedly pruned and then fine-tuned to recover lost accuracy, often achieving higher compression rates than a single pruning step.

Quantization reduces the numerical precision of a model's weights and activations, typically from 32-bit floating-point (FP32) to lower-bit formats like 16-bit (FP16/BF16), 8-bit integers (INT8), or even 4-bit integers (INT4). This drastically reduces memory bandwidth and enables faster computation on hardware that supports low-precision arithmetic.

• Post-Training Quantization (PTQ): Converts a pre-trained model to lower precision with minimal data for calibration. Fast but can lead to accuracy loss, especially for aggressive quantization (e.g., to INT8).
• Quantization-Aware Training (QAT): The model is trained or fine-tuned with simulated quantization noise, allowing it to learn to compensate for the precision loss. This yields higher accuracy for low-bit models (e.g., INT8) compared to PTQ.
• Dynamic Quantization: Weights are quantized ahead of time, but activations are quantized on-the-fly during inference. Useful for models like LSTMs where activation ranges vary significantly.

Knowledge Distillation (KD) is a training paradigm where a smaller, more efficient student model is trained to mimic the behavior of a larger, more accurate teacher model. The student learns not just from the hard labels (ground truth) but from the teacher's softened output probabilities (logits), which contain richer, inter-class relational information.

• Logit-based Distillation: The student's loss function combines a term for matching ground truth labels and a term for matching the teacher's softened logits, using a temperature parameter to control the softness of the probability distribution.
• Feature-based Distillation: The student is also trained to match intermediate representations or attention maps from the teacher's layers, providing a stronger learning signal.
• Self-Distillation: A variant where the teacher and student are the same model architecture, often at different stages of training, used as a regularization technique.

Low-rank factorization approximates large, dense weight matrices (common in fully-connected and attention layers) as the product of two or more smaller matrices. This exploits the intrinsic low-rank structure of many learned representations to reduce the number of parameters and FLOPs.

• Singular Value Decomposition (SVD): A mathematical technique that decomposes a matrix W into U, Σ, and Vᵀ. By keeping only the top-k singular values (from Σ), the matrix can be approximated as the product of two smaller matrices, significantly reducing parameters.
• Tensor-Train Decomposition: A more advanced factorization method for high-dimensional tensors, representing them as a chain of smaller core tensors. This is particularly effective for compressing the large embedding layers in recommendation models.
• The compressed model requires retraining or fine-tuning to recover accuracy after the approximation.

Weight sharing forces different parts of a model to use the same set of parameters, dramatically reducing the total parameter count. This is a form of extreme regularization that enforces structural priors on the model.

• Recurrent Neural Networks (RNNs): The canonical example, where the same transition weights are applied at every time step.
• Convolutional Neural Networks (CNNs): Use weight sharing implicitly via convolutional filters that slide across the input.
• Advanced Architectures: Techniques like ALBERT (A Lite BERT) apply cross-layer parameter sharing across all transformer layers, drastically shrinking the model size at the cost of some capacity.
• Differentiable Neural Architecture Search (DNAS): Can discover architectures where certain operations share underlying weights.

Enables the execution of sophisticated models directly on resource-constrained devices like smartphones, IoT sensors, and embedded systems. This eliminates cloud dependency, reduces latency for real-time applications (e.g., live translation, on-device photography), and enhances user privacy by keeping data local.

• Key Constraint: Limited memory (RAM/ROM), battery life, and CPU/GPU compute.
• Primary Techniques: Post-training quantization to 8-bit or lower, pruning to remove redundant weights, and specialized neural network compilation for mobile NPUs (e.g., Apple Neural Engine, Qualcomm Hexagon).
• Example: Deploying a vision transformer for object detection on a drone using TensorFlow Lite with full integer quantization.

Lowers the operational expense and improves responsiveness of models served at scale in data centers. Smaller models require less GPU/TPU memory and compute, allowing for higher throughput (requests per second) and lower latency per request.

• Key Metric: Cost per 1,000 inferences (CPI) and p95/p99 latency.
• Primary Techniques: Quantization (FP16, INT8) to accelerate tensor operations on hardware like NVIDIA Tensor Cores. Pruning and knowledge distillation to create smaller, faster student models.
• Impact: A 4x model size reduction can lead to proportional decreases in cloud compute costs and enable more efficient continuous batching of inference requests.

Achieves the strict latency budgets required for interactive systems. This is critical for applications where delay is unacceptable, such as autonomous vehicles, industrial robotics, augmented reality, and high-frequency algorithmic trading.

• Key Constraint: End-to-end latency must often be <100 milliseconds.
• Primary Techniques: Pruning and quantization to reduce computational graph complexity and accelerate kernel execution. Hardware-aware neural architecture search to design inherently efficient models.
• Example: A compressed LiDAR point cloud segmentation model running at 30 Hz on an automotive-grade NVIDIA Orin system-on-chip.

Allows large models to fit within the fixed memory (VRAM) of available accelerators. This is essential for fine-tuning or running inference with state-of-the-art models whose uncompressed size exceeds the capacity of standard GPUs.

• Key Constraint: GPU VRAM (e.g., 16GB, 24GB, 80GB).
• Primary Techniques: Quantization (e.g., loading a model in 4-bit precision via GPTQ or AWQ) to dramatically reduce memory footprint. Gradient checkpointing (a form of compression-for-training) trades compute for memory during backpropagation.
• Example: Loading a 70B parameter language model for inference on a single 24GB GPU using 4-bit quantization, which would otherwise require >140GB of FP16 memory.

Facilitates decentralized learning paradigms by reducing the communication overhead and on-device compute burden. In federated learning, models are trained across thousands of edge devices; compressed models make this process feasible.

• Key Constraint: Limited device compute, bandwidth for model update transmission.
• Primary Techniques: Structured pruning creates sparse models with smaller update sizes. Quantization reduces the bit-width of gradient updates sent to the central server.
• Benefit: Enables practical federated edge learning in regulated industries (healthcare, finance) by minimizing the resource impact on user devices while preserving data privacy.

Represents the extreme frontier of compression, enabling basic ML models to run on microcontrollers (MCUs) with kilobytes of memory and milliwatt power budgets. This unlocks AI in ubiquitous, low-cost sensors and wearables.

• Key Constraint: < 1 MB of SRAM/Flash, ultra-low power consumption.
• Primary Techniques: Extreme quantization (e.g., to 8-bit or binary/ternary weights), aggressive pruning (sparsity >90%), and manual operator substitution (e.g., replacing softmax with a simpler function).
• Framework: TensorFlow Lite for Microcontrollers, which uses a flatbuffer format and supports quantized kernels for ARM Cortex-M series processors.

Pruning is the systematic removal of parameters (weights, neurons, or entire layers) from a neural network that contribute minimally to its output. This creates a sparser, more efficient model.

• Structured Pruning: Removes entire structural components, like channels or filters, for direct hardware acceleration.
• Unstructured Pruning: Targets individual weights with low magnitude, creating an irregular, sparse pattern that requires specialized libraries for speedup.
• Magnitude-based Pruning is a common method where weights below a certain threshold are zeroed out, often followed by fine-tuning to recover accuracy.

Quantization reduces the numerical precision of a model's weights and activations, typically from 32-bit floating-point (FP32) to lower bit-widths like 16-bit (FP16), 8-bit integers (INT8), or even 4-bit. This drastically cuts memory use and accelerates computation on hardware that supports low-precision math.

• Post-Training Quantization (PTQ): Applied after training; faster but may cause accuracy loss.
• Quantization-Aware Training (QAT): Simulates quantization during training, allowing the model to adapt and preserve higher accuracy.
• Key trade-off is between the precision loss and the gains in inference speed and model size.

Knowledge Distillation is a training process where a smaller, more efficient student model is trained to mimic the behavior of a larger, more accurate teacher model. The student learns not just from the hard labels (ground truth) but from the teacher's softened output probabilities (logits), which contain richer relational information.

• Transfers the generalization capability of a large model into a compact form.
• The distillation loss function typically combines a term for matching the teacher's predictions with the standard cross-entropy loss for the true labels.

Low-Rank Factorization approximates large weight matrices in a neural network (common in dense and attention layers) as the product of two or more smaller matrices. This exploits the idea that the learned representations often lie in a lower-dimensional subspace.

• For a weight matrix W (m x n), it is approximated as W ≈ U (m x r) * V (r x n) where r (the rank) is much smaller than m or n.
• This reduces the number of parameters from m*n to r*(m+n), yielding significant compression, especially for wide layers.

Weight Sharing is a compression technique where multiple connections in a network use the same parameter value. This drastically reduces the number of unique parameters that need to be stored.

• Hashing Trick: Uses a hash function to map a large number of potential connections into a much smaller fixed-size bucket of weights.
• Fundamental to architectures like Convolutional Neural Networks (CNNs), where the same filter (shared weights) is slid across the input.
• In transformers, techniques like Albert use cross-layer parameter sharing across the encoder blocks.

Neural Architecture Search (NAS) automates the design of neural network architectures. For compression, Hardware-Aware NAS directly searches for model structures that achieve an optimal trade-off between accuracy and efficiency metrics (e.g., latency, FLOPs, memory) on a target device.

• The search space is constrained to include efficient operations like depthwise separable convolutions or bottleneck layers.
• It produces inherently efficient models, such as MobileNet and EfficientNet families, without requiring post-hoc compression.