Performance Baseline

The primary measure of a model's correctness on its intended task, establishing the upper bound for compressed variants. This is the most critical reference metric.

• Key Metrics: Task-specific scores like Top-1/Top-5 accuracy for classification, BLEU/ROUGE for translation/summarization, or mean Average Precision (mAP) for object detection.
• Validation vs. Test Set: The baseline is typically established on a held-out validation set to guide compression, with final reporting on a separate test set to ensure generalization.
• Golden Standard: This accuracy defines the golden model performance. Any deviation in compressed models is measured as accuracy drop or output divergence.

The storage footprint and structural composition of the original model, serving as the reference for compression ratio calculations.

• Parameter Count: The total number of trainable weights (e.g., 175B parameters for GPT-3). This directly influences memory requirements.
• Disk Footprint: The size of the serialized model file in megabytes or gigabytes, determined by parameter count and precision (typically FP32 or FP16).
• Architectural Details: The specific configuration of layers, attention heads, hidden dimensions, and activation functions. This structural blueprint is essential for planning layer-wise sensitivity analysis and mixed-precision quantization.

The theoretical floating-point operations required for a single forward pass, indicating the raw computational intensity of the model.

• FLOPs Definition: Measures the model's inherent compute demand, independent of hardware. A 1-billion-parameter dense layer requires ~2 GFLOPs for a single inference.
• Baseline for Sparsity: For techniques like model pruning, the baseline FLOPs count is used to calculate the theoretical speedup from creating a sparse model.
• Limitation: FLOPs are a hardware-agnostic metric and do not directly correlate with real-world latency, which is heavily influenced by memory bandwidth and hardware optimization.

The time required to execute a single forward pass of the model, measured in milliseconds under controlled conditions. This is the user-facing performance metric.

• Measurement Context: Must be profiled on a reference hardware platform (e.g., a specific CPU, GPU, or NPU) with a fixed batch size (often batch=1 for edge scenarios).
• End-to-End vs. Model-Only: Can refer strictly to model execution or include pre/post-processing. The baseline must clearly define the scope.
• Primary Target for Compression: Inference optimization techniques like quantization directly target reducing this latency. The baseline establishes the degradation threshold for acceptable speed-accuracy tradeoffs.

The pattern and volume of memory accesses during inference, which often becomes the true bottleneck for latency, especially on edge devices.

• Weights & Activations: The baseline profile quantifies the size of weights loaded from DRAM and intermediate activations stored in cache/RAM.
• Impact of Compression: Quantization directly reduces the bandwidth needed for weights. Pruning can create irregular access patterns that may hurt performance without specialized sparse inference kernels.
• Hardware-Aware Metric: Critical for hardware-aware compression and optimizing for specific memory hierarchies in mobile SoCs or NPUs.

The energy used per inference, a critical baseline for battery-constrained edge AI and tinyML deployments.

• Measurement: Typically measured in millijoules (mJ) per inference or watts (W) of average power draw during sustained operation.
• Direct Drivers: Correlates strongly with computational cost (FLOPs) and memory bandwidth, as data movement is often more expensive than computation.
• Tradeoff Analysis: Establishes the baseline for energy-efficient inference. Compression aims to drastically reduce this metric, sometimes at the cost of a controlled accuracy drop.

The fundamental engineering compromise in model compression where reductions in model size, latency, or memory footprint are balanced against potential decreases in predictive performance. This tradeoff is visualized on a tradeoff curve, where the goal is to find configurations on the Pareto frontier—points where no further compression can be achieved without unacceptable accuracy loss.

Accuracy Drop is the measurable decrease in a model's validation accuracy after compression. The Degradation Threshold (or Acceptable Loss) is the predefined, application-specific maximum allowable drop. For example, a real-time object detector might tolerate a 2% drop in mAP, while a medical diagnostic model may have a threshold of <0.5%. Exceeding this threshold renders the compressed model unfit for deployment.

Quantization Error is the numerical discrepancy from converting floating-point values to integers, a primary source of accuracy loss. This error manifests as Compression Artifacts—predictable distortions in model output. In vision models, this can cause misclassified edge cases; in language models, it may increase hallucination rates. Quantization Noise is the statistical modeling of this error as an additive signal perturbing weights and activations.

Sensitivity Analysis is a systematic method to identify which network components are most critical for accuracy. Layer-Wise Sensitivity measures the accuracy impact of compressing individual layers. This analysis directly informs Mixed-Precision Quantization strategies, where sensitive layers (e.g., attention heads in a transformer) are kept at higher precision (8-bit), while less sensitive layers (e.g., certain feed-forward networks) are aggressively quantized to 4-bit.

The process of regaining performance lost during compression. Fine-Tuning After Compression is the standard technique: the compressed model is further trained on a task-specific dataset. This allows the model to adapt to the introduced quantization noise or structural sparsity. More advanced methods include Quantization-Aware Training (QAT), which simulates quantization during training for better final accuracy.

Model Fidelity measures how closely a compressed model's behavior matches the original. It's assessed beyond simple accuracy, using metrics like KL Divergence to compare output probability distributions. Output Divergence is the general term for when predictions or internal activations drift. High fidelity is crucial for applications where the compressed model must be a drop-in replacement, ensuring downstream systems behave identically.