Compression Policy

The policy begins by defining quantitative goals, which serve as constraints for the optimization process. These are not arbitrary but are derived from deployment hardware limits.

• Target Metrics: Specific goals for model size (e.g., <10MB), inference latency (e.g., <50ms), and energy consumption.
• Accuracy Budget: The maximum allowable degradation in key performance metrics (e.g., top-1 accuracy drop <2%).
• Hardware Profile: The policy is informed by the target hardware's capabilities, such as support for INT8 operations or sparse tensor cores.

This component specifies which compression methods to apply and in what order, as techniques can have synergistic or conflicting effects.

• Primary Techniques: The core methods, such as pruning for sparsity, quantization for lower precision, or knowledge distillation for architectural efficiency.
• Orchestration Schedule: A plan for sequential application (e.g., prune first, then quantize) or simultaneous co-application, as in Quantization-Aware Training (QAT) with sparsity.
• Technique Parameters: For each method, defines specific parameters like the pruning algorithm (magnitude, gradient-based), quantization scheme (per-tensor, per-channel), and bit-width (8-bit, 4-bit).

A neural network is not uniformly sensitive to compression. This analysis identifies which layers can be aggressively compressed and which require careful treatment.

• Sensitivity Metrics: Measures the impact on loss or accuracy when a layer is pruned or quantized. Layers with high sensitivity are compressed lightly or last.
• Non-Uniform Policy: Results in a layer-specific plan, e.g., applying 8-bit quantization to most layers but keeping attention outputs in FP16.
• Automated Analysis: Often performed using libraries or via a calibration dataset to profile activation ranges and Hessian information.

The temporal plan that governs the progression of compression intensity during training or fine-tuning. A poor schedule can lead to irreversible accuracy loss.

• Iterative vs. One-Shot: Chooses between gradual pruning over many epochs or one-shot pruning followed by fine-tuning.

• Progress Function: Defines how sparsity or quantization intensity increases. Common schedules include:

  • Linear: Simple, constant increase.
  • Cosine Annealing: Smooth, gradual ramp-up.
  • Adaptive: Rate adjusts based on real-time validation loss (feedback-driven scheduling).

• Recovery Periods: Mandates fine-tuning phases after compression steps to allow the network to recover accuracy.

A robust policy includes continuous monitoring and contingency rules to prevent the model from degrading beyond the acceptable accuracy budget.

• Validation Checkpoints: The model is evaluated on a hold-out validation set at defined intervals during the compression process.
• Rollback Conditions: Predefined thresholds (e.g., accuracy drop >1% in a single phase) that trigger a rollback to a previous checkpoint and an adjustment of the policy (e.g., reducing the pruning rate).
• Pareto Frontier Tracking: The policy may aim to discover optimal points on the compression-accuracy Pareto frontier, stopping when no better trade-off can be found.

Modern compression policies are rarely implemented manually. This component specifies the software frameworks and automation strategies to execute the policy reliably.

• Automated Frameworks: Leverages tools like Neural Magic's SparseML, TensorFlow Model Optimization Toolkit, or PyTorch's Torch.ao to apply scheduled pruning and QAT.
• Search-Based Automation: Employs Automated Model Compression (AMC) or Hardware-Aware Neural Architecture Search (HW-NAS) to use reinforcement learning to discover the optimal layer-wise policy.
• Reproducibility: Ensures the policy is defined in code (e.g., YAML configuration files) and can be version-controlled, making the compression process reproducible and auditable.

A static schedule predefines the timing and intensity of compression actions before training begins. This is the most common and deterministic approach.

• Examples: A schedule that prunes 20% of weights every 10 epochs, or a plan to switch from FP32 to INT8 quantization at epoch 50.
• Advantages: Simple to implement, reproducible, and easy to debug.
• Use Case: Standard model optimization pipelines where the target sparsity or quantization level is known in advance.

An adaptive schedule dynamically adjusts compression intensity based on real-time training feedback, such as loss or gradient metrics.

• Mechanism: The policy uses a controller (often a simple heuristic or a learned agent) to increase the pruning rate when validation loss is stable or decrease it during periods of high loss.
• Goal: To prevent accuracy collapse by responding to the model's current state, potentially achieving a better accuracy-compression trade-off.
• Example: Increasing sparsity more aggressively when gradient magnitudes are small, indicating a stable optimization landscape.

This axis defines whether compression is applied in a single step or gradually over time.

• One-Shot Policy: Applies the full target compression (e.g., prune 50% of all weights) in a single operation, typically after training is complete. The compressed model is then fine-tuned. Simpler but can cause significant accuracy loss.
• Iterative Policy: Applies compression in many small steps interleaved with recovery phases. For example, Iterative Magnitude Pruning removes a small percentage of weights, fine-tunes, and repeats. This is guided by principles like the Lottery Ticket Hypothesis and generally preserves accuracy better.

A sensitivity-aware policy applies non-uniform compression across a model based on a layer-wise sensitivity analysis.

• Process: The policy profiles each layer to determine how much its compression degrades overall accuracy. Critical layers are compressed lightly or not at all, while robust layers are compressed aggressively.
• Outcome: Creates an optimized sparsity distribution or mixed-precision scheme, maximizing compression gains while minimizing accuracy loss.
• Tooling: Often implemented via Automated Model Compression (AMC) frameworks that use reinforcement learning to search for this optimal per-layer policy.

A multi-stage policy sequences different compression techniques in a specific order to achieve compound savings.

• Common Sequence: Pruning (to create sparsity) → Training/Fine-tuning (to recover accuracy) → Quantization (to reduce precision of remaining weights).
• Rationale: Pruning and quantization can interact. Pruning first removes redundant parameters, simplifying the model for subsequent quantization. Applying them together in one stage can be unstable.
• Advanced Variant: May include Knowledge Distillation as a final stage to further refine the small, quantized model using a larger teacher.

A hardware-aware policy co-designs the compression strategy with the target deployment hardware's characteristics.

• Objective: Maximize real-world metrics like latency, energy consumption, or memory bandwidth use, not just theoretical parameter count.
• Implementation: The policy uses a hardware cost model as a feedback signal. For example, it favors structured pruning (removing entire filters) over unstructured pruning if the target GPU executes structured sparsity faster.
• Search Method: Often found in Hardware-Aware Neural Architecture Search (HW-NAS), where the search algorithm directly optimizes for on-device latency or power.

A pruning schedule is a predefined strategy dictating the timing, rate, and criteria for removing parameters during training. It defines the progression from a dense to a sparse network.

• Key Phases: Often includes a warmup period, followed by iterative pruning and recovery fine-tuning.
• Objective: To reach a target sparsity ratio (e.g., 90% zeros) with minimal accuracy loss. Schedules can be one-shot (prune once) or gradual (prune incrementally).
• Example: A schedule might prune 20% of remaining smallest-magnitude weights every 10 epochs after epoch 50.

A QAT schedule outlines the phases for simulating quantization during training to recover accuracy. It governs when fake quantization nodes are inserted and how the model adapts.

• Typical Flow: Begins with a full-precision warmup, introduces quantization simulation, and concludes with calibration & fine-tuning.
• Critical Parameters: Includes scheduling the quantizer's bit-width and the steepness of the straight-through estimator (STE) used for gradient flow.
• Purpose: Ensures the model learns to be robust to the precision loss it will encounter during integer-only inference.

Automated Model Compression is a framework that uses reinforcement learning or search algorithms to automatically determine the optimal compression policy per layer.

• Process: An RL agent or controller network takes layer states as input and outputs actions like pruning ratio or quantization bit-width.
• Reward Signal: Typically a weighted sum of model accuracy and a hardware metric like latency or model size.
• Outcome: Discovers non-uniform policies that are more efficient than hand-designed, uniform schedules, often pushing the Pareto frontier of the accuracy-efficiency trade-off.

Layer-wise sensitivity measures how much a model's accuracy degrades when a specific layer is compressed. It is a foundational metric for designing non-uniform compression policies.

• Measurement: Often evaluated by applying compression to a single layer in isolation and observing the validation accuracy drop.
• Policy Impact: Guides the allocation of sparsity and precision. Highly sensitive layers (e.g., first and last) are compressed lightly, while robust middle layers can be aggressively compressed.
• Analysis Method: Can be determined via empirical profiling or estimated using second-order Hessian information.

Multi-stage compression is a scheduling paradigm where different techniques are applied in sequential, distinct phases, each followed by recovery fine-tuning.

• Common Sequence: Pruning → Fine-tune → Quantization → Fine-tune. This isolates the effects of each technique.
• Advantage: Simplifies the optimization landscape and makes the accuracy recovery process more stable compared to applying all techniques simultaneously.
• Use Case: The standard industrial pipeline for producing a heavily compressed model (e.g., a sparse, INT8 model for mobile deployment).

The Pareto frontier represents the set of optimal model configurations where no further compression can be achieved without sacrificing accuracy, and vice-versa.

• Visualization: A 2D plot with metrics like Model Size (MB) on the X-axis and Validation Accuracy (%) on the Y-axis. The frontier is the upper-left boundary of achievable points.
• Policy Goal: The objective of any compression schedule is to produce a model that lies on this frontier, maximizing efficiency for a given accuracy target.
• Analysis Tool: Used to compare the effectiveness of different pruning schedules or quantization policies.