Quantization

Applies different quantization bit-widths to different parts of a model based on their sensitivity to precision loss. This optimizes the trade-off between compression and accuracy.

• Principle: Not all layers contribute equally to error. Sensitive layers (e.g., attention outputs) are kept at higher precision (FP16), while robust layers (e.g., certain embeddings) are aggressively quantized (INT4).
• Automated Methods: Techniques like HAWQ (Hessian AWare Quantization) use the Hessian matrix to analytically determine layer sensitivity and assign optimal bit-widths.
• Result: Achieves a better accuracy-to-compression ratio than uniform quantization, creating a Pareto-optimal model for a target hardware constraint.

Techniques borrowed from information retrieval that are increasingly applied to compress embedding tables and large linear layers in neural networks.

• Vector Quantization (VQ): Groups weight vectors into clusters and replaces each vector with the index of its nearest cluster centroid (codeword). Storage shifts from weights to a small codebook and indices.
• Product Quantization (PQ): Splits a high-dimensional vector into subvectors and quantizes each sub-space independently. This creates a Cartesian product of codebooks, enabling efficient approximation of distances.
• Application: Highly effective for compressing the massive embedding tables in recommendation systems and the final classification layer in large vision models, where memory dominates.

Pruning is a neural network compression technique that systematically removes less important weights, neurons, or entire layers from a model. The goal is to create a sparser, smaller network that maintains accuracy.

• Process: Typically involves training a large model, evaluating parameter importance (e.g., via magnitude), removing low-importance elements, and fine-tuning.
• Types: Includes unstructured pruning (individual weights) and structured pruning (entire channels or filters).
• Use Case: Often combined with quantization in a compression pipeline to maximize model size reduction for edge deployment.

Knowledge Distillation is a model compression and training technique where a compact student model is trained to mimic the behavior of a larger, more accurate teacher model. The student learns from the teacher's output probabilities (soft labels) rather than just hard class labels.

• Mechanism: The soft labels provide a richer training signal, capturing the teacher's understanding of similarities between classes.
• Result: A smaller, faster model that can achieve performance close to the much larger teacher model.
• Application: Crucial for creating small language models (SLMs) and deploying models on resource-constrained devices.

Low-Rank Factorization is a model compression technique that approximates a large weight matrix (e.g., in a linear or convolutional layer) as the product of two or more smaller matrices. This exploits the idea that weight matrices often have a lower intrinsic rank than their dimensions suggest.

• Mathematical Basis: A weight matrix W of size m x n is approximated as W ≈ U * V, where U is m x r and V is r x n, with r (the rank) being much smaller than m or n.
• Benefit: Reduces the number of parameters from m*n to r*(m+n), decreasing memory and speeding up matrix multiplication.
• Common Use: Applied to fully connected and attention layers in transformers.

Embedding Compression refers to techniques for reducing the storage size and dimensionality of dense vector embeddings, which are fundamental to retrieval-augmented generation (RAG) and agentic memory systems.

• Methods: Includes quantization (e.g., 8-bit int embeddings), dimensionality reduction (e.g., PCA), and product quantization (splitting vectors into subvectors).
• Impact: Directly reduces the memory footprint of vector databases, enabling larger knowledge bases on the same hardware and faster similarity search.
• Trade-off: Managed balance between compression ratio and retrieval accuracy (recall).

Deep Compression is a seminal three-stage pipeline for extreme neural network compression, comprising pruning, quantization, and Huffman coding. Introduced by Han et al., it can reduce model size by 35x to 49x without loss of accuracy.

• Stage 1: Pruning: Removes redundant connections.
• Stage 2: Quantization: Clusters and shares weights across layers.
• Stage 3: Huffman Coding: Applies entropy coding to the quantized weights for further storage reduction.
• Significance: This holistic approach demonstrated the feasibility of deploying complex models like AlexNet on mobile and embedded hardware.

A Sparse Representation is a data format or model state where most elements are zero. Leveraging sparsity is a powerful compression strategy, as only non-zero values and their indices need to be stored and computed.

• In Models: Achieved via pruning, leading to sparse neural networks. Hardware support (e.g., NVIDIA's Sparsity SDK) accelerates sparse matrix operations.
• In Data: Natural in many domains (e.g., bag-of-words, transaction data). Formats like Compressed Sparse Row (CSR) efficiently store sparse matrices.
• Structured Sparsity: A hardware-aware variant where weights are pruned in specific patterns (e.g., 2:4 sparsity: 2 non-zeros per block of 4) to maximize inference speedup on supported accelerators.