Performance Profiling

The primary measure of a model's task performance, quantifying its ability to make correct predictions. This is the central metric balanced against compression gains.

• Key Metrics: Top-1 Accuracy, Top-5 Accuracy, F1-Score, Mean Average Precision (mAP).
• Measurement Context: Always measured on a held-out validation set distinct from the training and calibration data to ensure generalization.
• Impact of Compression: Techniques like quantization introduce quantization error, while pruning removes parameters, both potentially causing accuracy drop. The goal is to stay within an application's acceptable loss threshold.

The time required for a model to process a single input and produce an output, measured from start to finish. This is a critical user-facing metric for real-time applications.

• Measurement: Typically reported in milliseconds (ms) or frames per second (FPS), often as a p95 or p99 latency to account for tail performance.
• Primary Drivers: Model complexity (FLOPs), memory bandwidth, and the efficiency of sparse model inference kernels on the target hardware.
• Compression Effect: Quantization accelerates compute by using integer arithmetic, while pruning reduces FLOPs. True gains are only validated via on-device evaluation.

The total amount of memory (RAM) required to store the model's parameters and intermediate activations during inference. This determines if a model can fit on constrained edge devices.

• Components: Model Weights (persistent storage) and Runtime Activations (temporary memory).
• Key Metric: Compression Ratio, calculated as (Original Model Size) / (Compressed Model Size).
• Optimization Techniques: Post-training quantization reduces weight precision from 32-bit floats to 8-bit integers (4x size reduction). Pruning creates sparse weights, which can be stored efficiently in specialized on-device model formats like TFLite or ONNX.

The electrical energy used by the hardware to execute the model inference. This is paramount for battery-powered IoT and mobile devices.

• Measurement: Reported in milliwatt-hours (mWh) or Joules per inference.
• Correlation: Heavily tied to inference latency and memory bandwidth; reducing data movement is often more impactful for energy-efficient inference than reducing compute.
• Hardware Awareness: Profiling power requires hardware-aware compression strategies. Techniques like weight clustering can improve cache locality, significantly reducing dynamic power draw on mobile SoCs and NPUs.

Measures how closely a compressed model's behavior matches the original, uncompressed golden model, beyond simple task accuracy.

• Purpose: Detects subtle compression artifacts and changes in output distribution that might affect downstream systems.
• Key Metrics:
  • KL Divergence: Quantifies the difference between the output probability distributions of the original and compressed models.
  • Cosine Similarity: Measures the alignment of embedding vectors or internal activations.
• Use Case: Essential for ensuring a distilled student model reliably mimics its teacher model, or that a quantized model's confidence scores remain calibrated.

Profiles how effectively the model's computation maps to and utilizes the underlying silicon resources, such as CPU cores, GPU streams, or NPU compute units.

• Key Indicators: Compute Unit occupancy, cache hit/miss rates, memory bandwidth saturation.
• Goal of Optimization: To expose parallelism and structure computations for the hardware via compute graph optimization (e.g., operator fusion, kernel tuning).
• Connection to Compression: Sparsity patterns from pruning must align with hardware support for sparse operations. Mixed-precision quantization can leverage specialized low-precision units (e.g., INT8 tensor cores) for maximum throughput.

The original, uncompressed model's key metrics (accuracy, latency, memory footprint, power draw) used as the definitive reference point. All compression gains and losses are measured as deltas from this baseline. Establishing a robust baseline is the first step in any profiling workflow.

• Golden Model: The high-accuracy, uncompressed reference against which all compressed variants are compared for fidelity.
• Validation Set: The held-out dataset used for baseline measurement to prevent data leakage and overfitting.

A standardized evaluation suite of models, datasets, and hardware targets used to objectively measure and compare compression techniques. It provides reproducible metrics for latency, throughput, memory usage, power consumption, and accuracy. Common examples include MLPerf Tiny and EEMBC MLMark.

• On-Device Evaluation: The final, critical benchmark stage executed on the actual target hardware (e.g., a specific mobile SoC or microcontroller) to capture real-world performance, including thermal throttling and memory bandwidth effects.

A systematic profiling method that determines which components of a neural network are most critical to performance. It measures the accuracy degradation when compression is applied to individual layers, channels, or operators.

• Layer-Wise Sensitivity: The quantified impact of compressing each network layer, used to guide mixed-precision quantization strategies where more sensitive layers retain higher bit-widths.
• Calibration Dataset: A small, representative dataset (distinct from training/validation sets) used during this analysis to profile activation ranges and guide quantization parameters.

The degree to which a compressed model's outputs match the original model's behavior. It is a broader measure than task accuracy, assessing the statistical similarity of internal representations and prediction distributions.

• KL Divergence (Kullback–Leibler): A core metric for fidelity, quantifying the difference between the output probability distributions of the original and compressed models.
• Output Divergence: The general phenomenon where predictions or activations deviate, which can be measured via cosine similarity or mean squared error (MSE) of logits.

The application-specific, predefined limit for acceptable performance loss. Also called Acceptable Loss, this is a business or product requirement that defines the viable Pareto frontier for compression.

• Example: A real-time object detection model for a drone may have a strict latency threshold of 30ms and an accuracy drop threshold of no more than 2% in mean Average Precision (mAP).
• This threshold turns profiling data into a go/no-go decision for deployment, separating technically interesting compression from production-viable compression.

Predictable, undesirable errors in model output caused by the lossy nature of compression algorithms. Profiling aims to detect and quantify these artifacts.

• Quantization Noise: The error signal modeled as additive noise from converting continuous values to discrete integer representations, a primary source of accuracy drop.
• Sparsity Pattern Errors: In pruning, inefficient runtime handling of sparse weights can lead to latency spikes or unpredictable memory access patterns, negating theoretical FLOPs reductions.