Automated Model Compression (AMC)

AMC treats each layer's compression parameters (e.g., pruning ratio, quantization bit-width) as a discrete action in a search space. It uses a controller (often an RNN or a lightweight policy network) to sequentially sample a compression policy for the entire model. The key innovation is recognizing that layer-wise sensitivity varies significantly; a uniform compression strategy is suboptimal. The controller learns to allocate more sparsity or aggressive quantization to robust layers while preserving critical ones.

The core search mechanism in AMC is typically Reinforcement Learning (RL). The controller is the agent, the compression action sequence is the policy, and the compressed model's accuracy and size on a validation set form the reward signal (e.g., Reward = Accuracy - λ * Model Size). Through policy gradient methods, the controller learns to generate policies that maximize this compound reward, effectively navigating the compression-accuracy Pareto frontier without manual heuristic design.

A defining feature of advanced AMC is the direct incorporation of hardware feedback into the optimization loop. The reward function is extended beyond mere parameter count to include real, measurable on-device metrics. This transforms the search into a Hardware-Aware Neural Architecture Search (HW-NAS) problem for compression. The reward can penalize latency, energy consumption, or memory bandwidth, leading to policies optimized for specific NPUs, mobile SoCs, or microcontrollers.

While pioneering AMC used RL, subsequent research introduced more efficient search paradigms:

• Differentiable AMC: Formulates the pruning ratio selection as a continuous, differentiable optimization problem using Gumbel-Softmax or architectural parameters, enabling gradient-based search.
• One-Shot AMC: Leverages weight-sharing techniques from NAS. A supernet containing all possible compressed sub-networks is trained once. The optimal compression policy is then found by rapidly evaluating sub-networks sampled from this supernet, drastically reducing search cost.

AMC frameworks are designed to orchestrate multiple compression techniques simultaneously. A single policy can dictate both structured pruning ratios for convolutional filters and mixed-precision quantization bit-widths (e.g., 8-bit, 4-bit, 2-bit) for each layer's weights and activations. This allows for a multi-stage compression outcome in a single, automated search process, creating a model that is both sparse and quantized, optimized holistically rather than via sequential, independent steps.

AMC automates the core engineering challenge of compression scheduling. It replaces:

• Manual pruning schedules (e.g., iterative magnitude pruning, cosine pruning schedule).
• Heuristic sparsity distribution based on sensitivity analysis.
• Trial-and-error tuning of quantization-aware training (QAT) schedules. By framing it as a search problem, AMC discovers non-intuitive, layer-specific policies that often outperform human-designed schedules, especially under complex, multi-objective hardware constraints.

Tools like ProxylessNAS, Once-for-All, and HAT extend AMC by co-designing efficient architectures with target hardware constraints, a paradigm known as Hardware-Aware Neural Architecture Search (HW-NAS).

• Key Features: These frameworks search for subnetworks that are optimal for specific latency, energy, or memory budgets on target devices (e.g., mobile CPUs, NPUs). They define the compression-accuracy Pareto frontier for a given hardware platform.
• Use Case: Using the Once-for-All network to train a large super-network once, then efficiently extract specialized sub-networks optimized for different edge device latency targets without retraining.

While not AMC frameworks per se, deployment runtimes are critical for validating and executing compressed models. They provide the final inference optimization and hardware acceleration.

• Key Features: TensorFlow Lite includes converters that apply post-training quantization (PTQ) and run sparse model inference. ONNX Runtime supports running models quantized via its quantization tools and applies graph optimizations like operator fusion.
• Use Case: Taking a model compressed via an AMC policy in PyTorch, exporting it to ONNX, and using ONNX Runtime's quantization tools to apply dynamic quantization for final CPU deployment.

A compression policy is the comprehensive set of rules and algorithms that govern which compression techniques to apply, when to apply them, and with what intensity across a model's lifecycle. AMC is an automated method for generating this policy.

• Components: Defines the technique (pruning, quantization), schedule (when/how much), and layer-wise sensitivity adjustments.
• Output: For pruning, a policy might specify: "Prune 50% of Conv1 weights at epoch 10, 70% of Conv2 at epoch 20, using magnitude criteria."
• Role in AMC: The policy is the solution that the reinforcement learning agent or search algorithm learns to produce.

Multi-stage compression is a scheduling paradigm where different compression techniques are applied in separate, sequential phases, often with recovery fine-tuning in between. AMC can be used to optimize the schedule within each stage or the transition between them.

• Typical Pipeline: Pruning → Fine-tune → Quantization → Fine-tune → Knowledge Distillation.
• Challenge: The order and intensity of stages significantly impact final accuracy and size.
• AMC's Role: Can automate the decision of the optimal pipeline: e.g., should quantization come before or after pruning for this specific model and dataset?

Feedback-driven scheduling is an adaptive approach where compression decisions are continuously adjusted during training based on live metrics. It represents a dynamic, closed-loop alternative to static, pre-defined schedules, and is a core principle in many AMC implementations.

• Feedback Signals: Uses validation loss, gradient norms, or layer-wise sensitivity metrics measured in real-time.
• Action: Based on feedback, the algorithm might increase the pruning rate for a robust layer or pause compression for a sensitive layer.
• Example: An AMC agent observing rising loss might reduce the pruning intensity for the next few training steps.