Model Compression

A compression technique where a compact student model is trained to replicate the behavior of a larger teacher model. The student learns from soft targets—the teacher's probability distribution—which encode rich inter-class similarity information known as dark knowledge. This transfers generalization capabilities that hard labels alone cannot provide.

A compression method that reduces the numerical precision of a model's weights and activations after training is complete. Common approaches include:

• INT8 quantization: Converting 32-bit floating-point weights to 8-bit integers
• Dynamic quantization: Quantizing weights ahead of time while activations are quantized on-the-fly during inference
• Calibration: Using a small representative dataset to determine optimal clipping ranges for each layer

A technique that removes redundant or low-magnitude parameters from a neural network to create a sparse architecture. Strategies include:

• Unstructured pruning: Zeroing out individual weights with the smallest absolute values
• Structured pruning: Removing entire neurons, channels, or attention heads
• Iterative magnitude pruning: Alternating between pruning and fine-tuning cycles to recover accuracy
• Lottery Ticket Hypothesis: The finding that dense networks contain sparse subnetworks capable of training to full accuracy in isolation

A compression approach that decomposes large weight matrices into products of smaller matrices using techniques like Singular Value Decomposition (SVD). By approximating a weight matrix W as the product of two low-rank matrices U × V, the parameter count drops significantly. This is particularly effective for fully-connected layers and has been extended to convolutional kernels via tensor decomposition methods like CP-decomposition.

An automated approach that uses search algorithms to discover compact model architectures optimized for a target hardware platform. NAS explores the design space of layer types, channel counts, and kernel sizes under latency and memory constraints. Techniques include:

• Hardware-aware NAS: Incorporating inference latency on target devices directly into the search objective
• Once-for-All Networks: Training a single large network from which specialized sub-networks are extracted for different deployment scenarios

A compression technique applied to the training data rather than the model itself. It synthesizes a small set of informative training samples—often just a few images per class—such that a model trained exclusively on this synthetic set achieves performance comparable to one trained on the full dataset. This dramatically reduces training cost and storage requirements while preserving the essential information content of the original data distribution.