Model Compression Ratio

The model compression ratio is calculated as the size of the original model divided by the size of the compressed model. It is a dimensionless number, often expressed as N:1 (e.g., 4:1).

• Formula: Compression Ratio = Original Model Size / Compressed Model Size
• Example: A 100 MB model reduced to 25 MB has a compression ratio of 4:1.
• This primary metric focuses on the storage footprint, which directly impacts deployment feasibility on devices with limited flash memory.

While the ratio measures storage reduction, its ultimate value is realized through correlated improvements in computational efficiency and inference latency. A high compression ratio often, but not always, leads to significant reductions in Floating-Point Operations (FLOPs) and faster execution.

• Structured techniques like channel pruning or using EfficientNet architectures reduce parameters and FLOPs in tandem.
• Unstructured pruning or weight clustering may achieve high storage ratios but require specialized sparse hardware or software to translate into latency gains.
• The effective speedup is determined by the hardware-software stack and the alignment of the compression method with the accelerator's capabilities (e.g., INT8 support for quantized models).

The compression ratio exists on a Pareto frontier with model accuracy. Aggressively increasing the ratio typically incurs a accuracy penalty.

• The compression-accuracy trade-off is non-linear; initial compression (e.g., mild post-training quantization) may have negligible impact, while extreme compression (e.g., model binarization) often causes significant drops.
• Techniques like quantization-aware training (QAT) and knowledge distillation are designed to shift this curve, achieving a better ratio for a given accuracy target by allowing the model to adapt during training.
• The optimal operating point is use-case dependent, balancing resource constraints with minimum acceptable performance.

Different compression methods achieve characteristic ratios and have distinct implications:

• Quantization: Moving from FP32 to INT8 typically yields a ~4:1 storage ratio. Moving to BFloat16 gives a 2:1 ratio.
• Pruning: Can achieve 2:1 to 10:1+ ratios, depending on sparsity target. Structured pruning yields lower but more hardware-friendly ratios.
• Knowledge Distillation: The ratio is defined by the size difference between the teacher model and the student model (e.g., a 100M parameter teacher to a 10M parameter student is a ~10:1 ratio).
• Low-Rank Factorization: Compression ratio depends on the chosen rank; can approximate weight matrices with 3:1 to 10:1 reductions.

A critical limitation of the basic ratio is that it only measures parameter storage. For real-world inference, the memory footprint of intermediate activations can be a dominant bottleneck, especially for vision models with high-resolution inputs.

• Activation compression techniques (e.g., activation quantization, sparsification) target this separate constraint.
• A model with a high parameter compression ratio may still have large activation memory, limiting its deployment on devices with tight RAM budgets.
• A complete edge deployment profile requires analyzing both model size (persistent storage) and peak memory usage (RAM during inference).

The theoretical compression ratio only translates to practical gains if the target hardware and software stack support the compressed format.

• INT8 ratios are fully realized on hardware with dedicated integer matrix multiplication units (e.g., many NPUs, GPUs).
• High unstructured sparsity ratios require runtimes with sparse kernel support to avoid processing zeros.
• Hardware-aware pruning and neural architecture search (NAS) explicitly optimize the compression strategy for the target platform's execution engine, ensuring the reported ratio correlates with measured latency reduction.
• The final metric of success is end-to-end latency and power consumption on the target device, not the ratio in isolation.

A core technique for achieving high compression ratios by reducing the numerical precision of a model's weights and activations. This directly shrinks the model size and accelerates inference.

• Post-Training Quantization (PTQ): Converts a pre-trained model to a lower precision (e.g., FP32 to INT8) using a calibration dataset, without retraining.
• Quantization-Aware Training (QAT): Trains or fine-tunes the model with simulated quantization, allowing it to learn robust parameters for subsequent low-precision deployment.
• Common Targets: INT8, FP16, and BFLOAT16 are standard formats for edge inference, offering a balance of size, speed, and accuracy.

A technique that removes redundant or less important parameters from a neural network to reduce its size and computational cost.

• Structured Pruning: Removes entire structural components like channels, filters, or layers. This produces a smaller, dense network that runs efficiently on standard hardware.
• Unstructured Pruning: Removes individual weights, creating an irregular, sparse pattern. This can achieve high theoretical compression but requires specialized software or hardware (like sparse tensor cores) for real speedups.
• Hardware-Aware Pruning: Guides the pruning strategy based on the target hardware's architecture to maximize actual inference latency improvements.

A compression paradigm where a smaller, efficient student model is trained to mimic the behavior of a larger, more accurate teacher model. The goal is to transfer the teacher's knowledge into a compact form.

• The student learns from the teacher's soft labels (output probability distributions), which contain more information than hard class labels.
• This technique is particularly effective for compressing large Transformer-based models (e.g., BERT) into smaller versions for edge deployment.
• It addresses the compression-accuracy trade-off by using the teacher as a guide to preserve performance.

An automated process for designing optimal neural network architectures under specific constraints like model size, latency, or FLOPs.

• Instead of compressing an existing large model, NAS discovers inherently efficient architectures from scratch.
• EfficientNet and MobileNet are famous model families born from NAS and manual design principles for mobile/edge use.
• It directly optimizes for metrics like compression ratio and inference latency by searching over a space of operations (e.g., depthwise convolutions) and connectivity patterns.

These are the primary performance metrics that model compression techniques aim to improve, beyond just the compression ratio.

• FLOPs (Floating Point Operations): The number of calculations required for one inference. Reducing FLOPs directly lowers compute latency and energy consumption.
• Memory Footprint: The total RAM needed to store the model's parameters, activations, and intermediate buffers. This is a critical constraint for devices with limited memory (e.g., microcontrollers).
• A high compression ratio reduces the parameter storage footprint, but activation memory during inference is also a major bottleneck, addressed by techniques like activation compression.

The fundamental relationship in model compression where aggressive techniques to reduce size and latency often result in decreased model accuracy or fidelity.

• The goal is to push the Pareto frontier, finding the optimal model that provides the best accuracy for a given size or latency budget.
• Techniques like knowledge distillation and quantization-aware training are designed specifically to mitigate this trade-off.
• Evaluation requires measuring both the compression ratio (or FLOPs reduction) and the change in accuracy on a target task to understand the true cost of compression.