Model Compression Ratio

Quantization reduces the numerical precision of a model's weights and activations, converting them from high-precision formats (e.g., 32-bit floating-point) to lower-precision formats (e.g., 8-bit integers). This directly shrinks the model's memory footprint and can accelerate inference on hardware with efficient integer arithmetic units.

• Key Types: Post-Training Quantization (PTQ), Quantization-Aware Training (QAT), Static, and Dynamic quantization.
• Primary Impact: Achieves a high compression ratio by reducing the bit-width of each parameter, often by 4x (FP32 to INT8) or more.

Pruning removes redundant or less important parameters from a neural network. By eliminating weights, neurons, or entire filters, it reduces the model's parameter count and the computations required during inference.

• Structured Pruning: Removes entire structural components (e.g., filters, channels), resulting in a smaller, dense model that runs efficiently on standard hardware.
• Unstructured Pruning: Removes individual weights based on magnitude, creating a highly sparse model that requires specialized libraries or hardware for speedup.
• Goal: To identify and eliminate parameters that contribute minimally to the model's output, maximizing the compression ratio for a given accuracy budget.

Knowledge Distillation is a training procedure for model compression. A large, accurate teacher model is used to train a smaller, more efficient student model. The student learns to mimic not just the teacher's final predictions (hard labels) but also its softened probability distributions and internal representations.

• Mechanism: Transfers the 'dark knowledge' embedded in the teacher's logits and intermediate feature maps.
• Outcome: The student model often achieves higher accuracy than if trained on data alone, allowing for a more aggressive compression ratio while maintaining performance.

Neural Architecture Search automates the design of efficient model architectures tailored for specific constraints like parameter count or latency. Instead of manually compressing a large model, NAS searches a space of possible operations and connections to discover a naturally efficient network from scratch.

• Approach: Uses reinforcement learning, evolutionary algorithms, or gradient-based methods to explore the architecture space.
• Relation to Compression: Designs models that are inherently small and fast, achieving an optimal baseline compression ratio relative to hand-designed architectures for the same task.

Low-Rank Factorization compresses weight matrices by exploiting the idea that they often contain redundant information. It approximates a large weight matrix as the product of two or more smaller matrices, reducing the total number of parameters.

• Mathematical Basis: Applies techniques like Singular Value Decomposition (SVD) or Tucker decomposition to tensors.
• Application: Particularly effective for compressing the large fully-connected layers found in older architectures and the linear projections within Transformer models.
• Compression Effect: Directly reduces the parameter count, contributing to the overall model compression ratio.

This refers to hand-designed neural network structures that are inherently parameter- and compute-efficient. These architectures achieve high performance with a low baseline size, making further compression more effective.

• Examples: MobileNet (uses depthwise separable convolutions), EfficientNet (uses compound scaling), and Transformer variants like MobileViT.
• Design Principles: Use of factorized convolutions, inverted bottlenecks, and efficient attention mechanisms.
• Role in Compression: Starting with an efficient architecture sets a favorable initial compression ratio, as the model is already optimized for size and speed before techniques like quantization are applied.