On-Device Evaluation

The primary measure of real-time responsiveness, defined as the time elapsed from submitting an input to receiving a final prediction. This is the most direct metric for user experience.

• Measured in: Milliseconds (ms) or microseconds (µs).
• Key Factors: Model architecture, degree of compression, hardware compute units (CPU, GPU, NPU), and memory bandwidth.
• Evaluation Method: Average latency over thousands of inferences to account for system jitter and thermal throttling.

The rate of processing, defined as the number of inferences a system can complete per second. Throughput is critical for batch processing or serving multiple concurrent requests.

• Measured in: Frames Per Second (FPS) or inferences per second (inf/sec).
• Optimization Target: Maximizing throughput often involves techniques like batch processing, which trades off single-inference latency for higher overall efficiency.
• Hardware Saturation: High throughput indicates efficient utilization of parallel compute resources like GPU shader cores or NPU tensor arrays.

The total electrical energy used by the hardware to execute the model, directly impacting battery life and thermal design in mobile and IoT devices.

• Measured in: Milliwatts (mW) or Joules per inference.
• Measurement Tools: Requires specialized hardware probes or onboard power management unit (PMU) telemetry.
• Dynamic vs. Static Power: Evaluation must account for the dynamic power of active computation and the static power of idle silicon, which compression can affect differently.

The total amount of RAM required to load and execute the model, including weights, activations, and intermediate buffers. This is a hard constraint for microcontrollers and low-end mobile SoCs.

• Measured in: Megabytes (MB) or Kilobytes (KB).
• Components: Model Weights (post-compression size), Activation Memory (size of intermediate tensors), and Runtime Overhead (framework libraries).
• Peak Memory: The critical metric is the peak memory usage during inference, which determines the minimum required RAM on the target device.

The compressed model's performance on its primary task, measured against the original model's validation accuracy. This quantifies the accuracy drop from compression.

• Standard Metrics: Top-1/Top-5 Accuracy (classification), mAP (object detection), BLEU/ROUGE (language tasks).
• On-Device vs. Server Baseline: Accuracy must be measured on the device using the same quantized/runtime path as production to capture all numerical errors.
• Acceptable Loss Threshold: Defined per application (e.g., <1% drop for a vision model, <5% for a keyword spotter).

Measures how closely the compressed model's behavior matches the original, uncompressed golden model. This goes beyond task accuracy to evaluate internal consistency.

• Key Metrics: KL Divergence (difference in output probability distributions), Cosine Similarity (of embedding vectors), or Mean Squared Error (of logits or activations).
• Purpose: High fidelity ensures downstream systems relying on model embeddings or confidence scores continue to function correctly.
• Detection of Artifacts: Helps identify systematic compression artifacts or quantization noise that cause erratic behavior on specific input types.

The fundamental engineering compromise in model compression where reductions in model size, latency, or memory footprint are balanced against potential decreases in predictive performance. This is visualized using a tradeoff curve, where the goal is to find configurations on the Pareto frontier—points where you cannot improve one metric without worsening another. Determining the acceptable loss or degradation threshold is an application-specific business decision.

Accuracy Drop is the measurable decrease in a model's validation accuracy after compression. Accuracy Recovery is the process of regaining this lost performance, primarily through fine-tuning after compression (also called quantization-aware training for quantized models). The golden model (original, uncompressed) serves as the performance baseline for measuring this drop and the target for recovery efforts.

Quantization Error is the numerical discrepancy introduced when converting continuous floating-point values to discrete integer representations. This error is often modeled as quantization noise—additive noise that perturbs weights and activations. This noise is a primary source of model degradation and output divergence, leading to potential compression artifacts in the model's predictions.

Sensitivity Analysis systematically evaluates which parts of a network are most critical to preserve. Layer-wise sensitivity measures degradation when compressing individual layers, guiding strategies like mixed-precision quantization and bit-width selection. Robustness Analysis evaluates how compression affects performance on challenging inputs like out-of-distribution data or adversarial examples, ensuring the compressed model remains reliable.

Metrics that quantify how closely a compressed model's behavior matches the original. KL Divergence (Kullback–Leibler divergence) measures the difference between the output probability distributions of the original and compressed models. Model Fidelity is a broader term for this alignment, often also assessed via cosine similarity of embeddings or direct label agreement on a validation set.

The standardized practice of evaluating compression techniques. A compression benchmark uses fixed models, datasets, and metrics for objective comparison. Performance Profiling is the act of measuring key metrics—accuracy, latency, memory, power—before and after compression. The compression ratio (e.g., 4x size reduction) is a core quantitative output of this benchmarking process.