Layer-Wise Sensitivity

Compression does not affect all layers equally. Layer-wise sensitivity reveals that certain layers, often those with high-dimensional feature representations or critical for task-specific reasoning, suffer significant accuracy drop when quantized or pruned, while others are remarkably robust. For example, in a convolutional neural network, the final fully-connected classification layers are typically more sensitive than early convolutional filters. This non-uniformity is the fundamental rationale for moving beyond uniform compression to mixed-precision quantization.

Sensitivity analysis directly informs bit-width selection. By profiling the quantization error or output divergence for each layer, engineers can construct a heterogeneous compression plan:

• Assign higher precision (e.g., FP16, INT8) to sensitive layers.
• Aggressively compress robust layers to lower bit-widths (e.g., INT4). This data-driven approach allows for maximizing the overall compression ratio while staying within an acceptable loss threshold, creating a model on the Pareto frontier of the size-accuracy tradeoff curve.

A common proxy for measuring sensitivity is analyzing the Taylor expansion of the loss function with respect to layer parameters. Layers where the removal or perturbation of parameters causes a large predicted increase in loss are deemed highly sensitive. In practice, this is often estimated by looking at the average magnitude of gradients during inference on a calibration dataset or the L2-norm of the weights themselves. These heuristics provide a computationally efficient way to rank layers without exhaustive re-training.

There is no universal sensitivity map. The sensitivity profile is intrinsically linked to:

• Model Architecture: Transformers, CNNs, and RNNs exhibit fundamentally different sensitivity patterns.
• Task Complexity: Layers crucial for fine-grained discrimination in image segmentation will be more sensitive than those in a simpler classification task.
• Training Dataset: The data distribution influences which features and, consequently, which layers are most important. Therefore, sensitivity analysis must be performed anew for each golden model and target application.

Sensitivity analysis is the core engine of modern hardware-aware compression tools. Frameworks like TensorFlow Lite's Model Optimization Toolkit and NVIDIA's TensorRT use automated sensitivity profiling to:

1. Profile each layer's contribution to the overall loss.
2. Apply an optimization algorithm to allocate precision across layers.
3. Generate a compressed on-device model format (e.g., a .tflite file with mixed precision). This automation is essential for scaling compression across diverse model zoos and hardware targets.

The ultimate test of a sensitivity-based compression strategy is on-device evaluation. A profile that looks optimal in simulation may fail on real hardware due to:

• Inefficient kernel support for specific mixed-precision operations.
• Unexpected memory bandwidth bottlenecks.
• Quantization noise interacting non-linearly across layers. Final performance profiling on the target chip (e.g., a mobile NPU) is necessary to validate that the theoretical sensitivity map translates to real-world energy-efficient inference and meets the degradation threshold.

Beyond frameworks, engineers often build custom profiling scripts to measure layer-wise sensitivity metrics critical for deployment.

• Hook-based activation capture: Inserting forward hooks in PyTorch or TensorFlow to extract and compare outputs (e.g., using Mean Squared Error or Cosine Similarity) between original and perturbed (quantized/pruned) layers.
• Energy and Latency Profiling: Using hardware-specific profilers (e.g., TensorRT, Qualcomm SNPE) to measure the real inference latency and power consumption impact of compressing individual layers on the target silicon.
• Sensitivity Heatmaps: Creating visualizations that map accuracy degradation or output divergence across the model's depth, guiding decisions for heterogeneous compression strategies.

A systematic evaluation to determine which components of a neural network are most critical for preserving accuracy when compressed. This involves perturbing individual layers, channels, or parameter groups and measuring the resultant accuracy drop.

• Purpose: Guides heterogeneous compression strategies by identifying which parts of the model require higher precision.
• Method: Often performed via iterative pruning or simulated quantization of individual components while monitoring validation loss.
• Output: A sensitivity profile or ranking used to allocate bit-widths in mixed-precision quantization.

A quantization strategy that assigns different numerical precisions (bit-widths) to different tensors within a single model. This is a direct application of layer-wise sensitivity analysis.

• Core Idea: Apply aggressive quantization (e.g., 4-bit) to insensitive layers and conservative quantization (e.g., 8-bit or 16-bit) to sensitive layers.
• Benefit: Achieves a superior compression-accuracy tradeoff compared to uniform quantization.
• Implementation: Requires hardware support for multiple data types (e.g., INT4, INT8) within a single inference pass.

The numerical discrepancy introduced when converting continuous floating-point values to discrete integer representations. This error is a primary source of accuracy degradation and its impact varies layer-by-layer.

• Sources: Rounding error from mapping float to integer and clipping error from values outside the quantization range.
• Propagation: Error accumulates through forward passes; layers with high dynamic range or non-linear activations are often more sensitive.
• Mitigation: Techniques like quantization-aware training model this error during training to improve robustness.

The degree to which a compressed model's outputs and internal behaviors match those of the original, uncompressed golden model. It is a broader measure than just task accuracy.

• Metrics: Measured using KL Divergence between output distributions, cosine similarity of embeddings, or Mean Squared Error of intermediate activations.
• Importance: High fidelity ensures downstream systems relying on model embeddings or features remain functional.
• Layer-Wise Fidelity: Critical layers for task performance must maintain high fidelity, while others can tolerate more divergence.

The process of regaining performance lost during compression, typically through post-compression training. The required recovery effort is directly tied to layer-wise sensitivity.

• Primary Technique: Fine-tuning after compression, where the compressed model is trained for a few epochs on the original dataset.
• Advanced Methods: Quantization-aware training simulates quantization noise during training, making the model inherently more robust.
• Sensitive Layers: Layers identified as highly sensitive often require more training iterations or a lower learning rate during recovery.

The systematic measurement of a model's key metrics before and after compression to quantify the tradeoff. Profiling must be done per-layer to inform sensitivity analysis.

• Pre-Compression Baseline: Establishes the Golden Model's accuracy, latency, and memory footprint.
• Post-Compression Metrics: Measures accuracy drop, inference latency, and memory reduction.
• Granular Analysis: Modern profilers (e.g., TensorFlow Profiler, PyTorch Profiler) can attribute latency and memory usage to individual operators, enabling layer-wise optimization decisions.