Adaptive Compression

The defining mechanism of adaptive compression is a closed-loop system that continuously monitors model performance. Key metrics like validation loss, training loss, or task-specific accuracy are used as feedback signals. Based on predefined thresholds or a controller algorithm, the system dynamically adjusts compression parameters, such as:

• Pruning rate: Increasing or decreasing the percentage of weights removed.
• Quantization bit-width: Switching between precision levels (e.g., 8-bit to 4-bit).
• Sparsity pattern: Altering which structures (filters, channels) are targeted. This real-time adjustment prevents accuracy collapse and optimizes the trade-off between model size and performance.

Unlike static schedules, adaptive compression employs a policy—a set of rules or a learned function—that maps observed performance states to compression actions. This policy can be:

• Rule-based: "If validation accuracy drops by >0.5%, reduce the pruning rate by 10%."
• Learned: Using Reinforcement Learning (RL) where an agent learns to take compression actions that maximize a reward function balancing compression gains and accuracy.
• Gradient-informed: Adjusting compression based on the magnitude or distribution of gradients, which indicate parameter importance. The policy allows the system to respond to the unique learning dynamics of each model and dataset.

Adaptive compression recognizes that neural network layers have different sensitivities to compression. It applies non-uniform strategies across the network:

• Sensitive layers (e.g., final classification layers) may receive lighter compression or later application.
• Robust layers (e.g., early convolutional filters) can be compressed more aggressively and earlier. The system dynamically allocates a sparsity budget or precision budget across layers. This is often guided by real-time sensitivity analysis, such as monitoring the change in loss when perturbing a layer's weights, ensuring compression is applied where it is least harmful.

Adaptive compression is tightly coupled with the model's training lifecycle. It typically operates in phases:

1. Warm-up: Initial training with little to no compression to establish a stable model.
2. Adaptive Phase: Compression is introduced and adjusted based on feedback. Techniques like gradual pruning or quantization-aware training (QAT) are applied here.
3. Recovery/Fine-tuning: After a compression step, the model is given time to recover accuracy through continued training.
4. Convergence: Compression stabilizes as the model nears final performance targets. This phased approach prevents interference with critical early learning and allows the network to adapt to its changing architecture.

The system optimizes for multiple, often competing, objectives simultaneously. The primary trade-off is between model efficiency and model accuracy. Adaptive compression algorithms explicitly manage this by targeting a specific point on the Compression-Accuracy Pareto Frontier. Secondary objectives can include:

• Training speed: Compression can reduce forward/backward pass cost.
• Inference latency/power: Anticipating the final hardware deployment.
• Memory footprint: Constraining peak memory usage during training. The adaptation mechanism balances these objectives in real-time, making it superior to post-hoc compression for finding optimal deployment-ready models.

Adaptive compression is a core component of Automated Model Compression (AMC) and Hardware-Aware Neural Architecture Search (HW-NAS). In these frameworks:

• A controller (e.g., an RL agent) performs adaptive compression as a search over a space of possible compression actions.
• Each action (e.g., "prune 30% of layer 5's filters") is evaluated via a performance predictor or short training run.
• The feedback (accuracy, latency) is used to update the controller's policy. This creates a fully automated pipeline that discovers a tailored compression strategy for a given model, dataset, and hardware target, moving beyond hand-designed schedules.

This core adaptive technique uses live performance metrics to guide compression decisions in real-time. Instead of a fixed schedule, the algorithm monitors signals like validation loss, gradient norms, or task-specific accuracy to adjust parameters.

• Dynamic Adjustment: If accuracy drops beyond a threshold, the pruning rate is automatically reduced or paused.
• Gradient-Based Signals: Large gradient magnitudes in a layer can signal its importance, temporarily protecting it from aggressive compression.
• Enables Robustness: Allows the training process to recover from overly aggressive compression steps, leading to more stable convergence and better final accuracy compared to static schedules.

AMC is a reinforcement learning-based framework that automates the search for optimal layer-specific compression policies. It treats the compression strategy as a decision-making process.

• Agent-Environment Loop: An RL agent selects actions (e.g., prune 40% of layer 5's filters), observes the resulting model's accuracy and size on a validation set, and receives a reward.
• Policy Search: The agent learns a compression policy that maps model states (layer properties, current sparsity) to optimal actions.
• Hardware-Aware: Can incorporate latency or energy estimates into the reward function, making it a form of hardware-aware neural architecture search (HW-NAS) for compression.

This technique iteratively prunes and regrows network connections during training, allowing the sparse topology to evolve adaptively. It goes beyond simple removal.

• Two-Phase Cycle: In the pruning phase, connections with weights below a threshold are cut. In the regrowth phase, new connections are added where gradients suggest high potential.
• Maintains Fixed Sparsity: The total number of active connections is kept constant, exploring the sparse parameter space efficiently.
• Related to SET: Sparse Evolutionary Training (SET) is a prominent algorithm in this category, demonstrating that networks can be trained from a random sparse initialization with dynamic topology adjustments.

DNAS formulates architecture search, including compression decisions, as a continuous optimization problem solvable via gradient descent. It's used to find optimal sub-networks.

• Supernet and Path Weights: A supernet contains all possible candidate operations (e.g., different filter sizes, whether a layer is pruned). Each path has an associated architectural weight.
• Differentiable Selection: The search optimizes these architectural weights alongside model weights using a Gumbel-Softmax or similar trick, making the discrete choice of which path to use differentiable.
• Efficient Sub-Network Derivation: After training, the final compressed model is derived by selecting the paths with the highest architectural weights, yielding a model tailored for efficiency.

These are parametric, time-based strategies that increase sparsity smoothly, allowing the network to adapt. While not feedback-driven, their smoothness is a key adaptive principle.

• Gradual Pruning: Linearly or polynomially increases sparsity from an initial value (often 0%) to a final target over many epochs. This prevents shocking the network with a large, one-shot parameter removal.
• Cosine Pruning Schedule: Applies a cosine annealing function to the sparsity rate. The increase is very slow at the beginning and end of the schedule, with a steeper increase in the middle. This often leads to better accuracy recovery than linear schedules.
• Foundation for Adaptation: These predictable schedules are often used as a baseline upon which feedback-driven mechanisms are layered for finer control.

This technique applies non-uniform compression across the network based on each layer's sensitivity. It adapts the strategy to the model's internal structure.

• Sensitivity Analysis: Before or during training, the impact of compressing each layer is measured (e.g., accuracy drop from pruning 10% of its weights).
• Allocating Sparsity/Bit-width: A higher compression rate (more sparsity, lower bit-width) is applied to layers with low sensitivity. Critical layers are compressed lightly or not at all.
• Optimizing the Pareto Frontier: This approach directly targets the compression-accuracy Pareto frontier, finding the optimal trade-off point for each individual component rather than using a one-size-fits-all policy.

A compression policy is a comprehensive set of rules and algorithms that govern which techniques (e.g., pruning, quantization) to apply, when to apply them, and with what intensity across a model's lifecycle. It acts as the master plan, specifying:

• The sequence of compression stages.
• The target sparsity or bit-width for each layer.
• The conditions for transitioning between phases.
• The recovery mechanisms (e.g., fine-tuning epochs).

Adaptive compression is a type of policy where these rules are dynamically adjusted based on real-time feedback.

Feedback-driven scheduling is an adaptive approach where compression decisions—such as the pruning rate or quantization bit-width—are continuously adjusted based on live performance metrics. This is the core mechanism enabling adaptive compression.

Key feedback signals include:

• Validation loss and accuracy.
• Gradient norms and weight magnitudes.
• Hardware performance counters (e.g., latency, memory usage).

The scheduler uses this feedback in a control loop to decide whether to increase, decrease, or pause compression intensity to stay within an acceptable performance envelope.

Gradual pruning is a foundational scheduling strategy that incrementally increases a model's sparsity over many training steps or epochs, allowing the network to adapt smoothly. It is often a component within an adaptive framework.

A common implementation is the cosine pruning schedule, where sparsity is increased according to a cosine annealing function, providing a smooth ramp to the target. Adaptive compression can modulate this schedule in real-time—slowing the ramp if accuracy drops too quickly or accelerating it if the model is stable.

Layer-wise sensitivity is a critical metric that measures how much a model's accuracy degrades when a specific layer is compressed. It is used to guide non-uniform, adaptive compression policies.

In practice:

• Sensitivity is profiled using a small calibration dataset.
• Layers with low sensitivity (e.g., later convolutional layers) can be aggressively compressed.
• Layers with high sensitivity (e.g., attention output projections) are compressed lightly or last.

Adaptive schedulers may continuously re-evaluate sensitivity during training as the model's loss landscape evolves.

The compression-accuracy Pareto frontier defines the set of optimal model configurations where no further compression can be achieved without sacrificing accuracy, and vice-versa. Adaptive compression schedulers aim to navigate towards and along this frontier.

• The frontier is discovered through extensive profiling or search (e.g., AMC).
• It provides the theoretical limit for a given model and dataset.
• An adaptive scheduler's goal is to make runtime decisions that keep the model on this optimal curve, dynamically trading off size for accuracy based on current constraints or performance feedback.