Compression Benchmark

A benchmark's foundation is a curated set of reference models spanning architectures (e.g., ResNet, BERT, Vision Transformer) and complexities. This zoo provides a common baseline for comparison. Key aspects include:

• Diverse Architectures: Coverage of CNNs, Transformers, and RNNs.
• Pre-trained Weights: Publicly available, well-performing checkpoints.
• Task Variety: Models for image classification, object detection, NLP, and speech recognition.

Examples include models from PyTorch's TorchVision, Hugging Face's Transformers library, and TensorFlow's Model Garden.

Benchmarks require standardized datasets for two distinct phases:

• Calibration Dataset: A small, representative subset (e.g., 500-1000 samples) used during post-training quantization to estimate activation ranges. It must not overlap with evaluation data.
• Evaluation Dataset: A large, held-out validation or test set (e.g., ImageNet-1k val, GLUE benchmark) used to measure final accuracy metrics like Top-1, mAP, or F1-score. This ensures performance is measured on unseen data, preventing overfitting to the benchmark.

The definitive output of a benchmark is a suite of quantitative metrics that capture the compression-accuracy tradeoff. Core metrics include:

• Model Size: Measured in megabytes (MB) or parameter count, indicating storage and memory footprint reduction.
• Computational Cost: Measured in FLOPs (Floating Point Operations) or MACs (Multiply-Accumulate Operations).
• Inference Latency: End-to-end execution time (in milliseconds), ideally measured on target hardware.
• Accuracy: Primary task performance (e.g., classification accuracy).
• Compression Ratio: The ratio of original model size to compressed model size.

Advanced metrics may include energy consumption (joules per inference) and peak memory usage during inference.

For latency and power metrics to be meaningful, the benchmark must specify a controlled execution environment. This includes:

• Target Hardware: Specific CPUs (e.g., ARM Cortex-A78), GPUs, or NPUs (e.g., Google Edge TPU, Qualcomm Hexagon).
• Software Stack: Precise versions of the OS, drivers, inference frameworks (e.g., TensorFlow Lite, PyTorch Mobile, ONNX Runtime), and compiler flags.
• Execution Settings: Batch size (typically 1 for edge scenarios), number of warm-up runs, and number of measurement iterations.

This standardization prevents variance from software optimizations or thermal throttling from skewing results.

A benchmark evaluates specific, reproducible implementations of compression algorithms. Common categories include:

• Post-Training Quantization (PTQ): Converting FP32 models to INT8 using calibration data.
• Quantization-Aware Training (QAT): Fine-tuning models with simulated quantization.
• Pruning: Structured (removing entire filters/channels) or unstructured (removing individual weights) sparsity.
• Knowledge Distillation: Training a small student model using outputs from a large teacher.

The benchmark must document the exact hyperparameters, libraries (e.g., TensorFlow Model Optimization Toolkit, PyTorch FX Graph Mode Quantization), and procedures used for each technique.

The final component is a standardized format for presenting results, enabling clear comparison. This involves:

• Tradeoff Curves: 2D plots with accuracy (y-axis) versus model size or latency (x-axis), highlighting the Pareto frontier of optimal configurations.
• Summary Tables: Consolidated metrics for all model-technique combinations.
• Fidelity Metrics: Reporting of KL divergence or cosine similarity between original and compressed model outputs to measure behavioral fidelity beyond top-line accuracy.

Established benchmarks like MLPerf Tiny provide templates for this structured reporting, ensuring results are interpretable and actionable for engineers.

The fundamental engineering compromise where reductions in a model's size, latency, or memory footprint are balanced against potential decreases in its predictive accuracy. This is the core problem a compression benchmark seeks to quantify.

• Key Metrics: Benchmarks plot this relationship, creating a tradeoff curve.
• Design Goal: The ideal compression technique pushes this curve towards the Pareto frontier, offering maximal compression for minimal accuracy loss.

The original, uncompressed model's key performance metrics, serving as the absolute reference point for all compression evaluations. A benchmark is meaningless without this baseline.

• Golden Model: Often synonymous with the performance baseline model.
• Measured Metrics: Includes validation accuracy, inference latency, model size (parameters/MB), and FLOPs.
• Purpose: All reported compression ratios and accuracy drops are calculated relative to this baseline.

The primary visualization tool in compression benchmarking.

• Tradeoff Curve: A plot (e.g., accuracy vs. model size) showing all evaluated compression configurations. Each point represents a compressed model variant.
• Pareto Frontier: The subset of points on the curve that are Pareto optimal. For these models, you cannot improve one metric (e.g., accuracy) without worsening another (e.g., size). Benchmark rankings prioritize techniques that dominate the frontier.

The critical, final stage of benchmarking where compressed models are profiled on actual target hardware. This moves beyond theoretical metrics to measure real-world deployment viability.

• Measures: Real inference latency, memory bandwidth usage, power consumption, and thermal impact.
• Reality Check: A model that excels in a software-only benchmark may fail here due to lack of hardware-aware kernel support or inefficient memory access patterns.

Metrics that assess how closely a compressed model's behavior matches the original, beyond just task accuracy.

• Model Fidelity: The degree of output alignment. Measured by KL Divergence (for probability distributions) or cosine similarity (for embeddings/logits).
• Output Divergence: The general phenomenon of behavioral drift.
• Use Case: Critical for knowledge distillation evaluation and for ensuring downstream system stability when swapping a model.

The application-specific, business-defined constraints that determine if a benchmark result is a success or failure.

• Acceptable Loss: The maximum allowable drop in a key metric (e.g., ≤1% top-1 accuracy loss) for deployment.
• Degradation Threshold: The hard line defined by this loss.
• Benchmark Role: A good compression benchmark suite allows engineers to quickly filter results to find models that meet their specific deployment constraints.