Filter Pruning

Unlike unstructured pruning which creates irregular sparsity patterns, filter pruning removes entire convolutional filters. This results in structured sparsity—the pruned weight tensors remain dense matrices, allowing the compressed model to run efficiently on standard hardware (CPUs/GPUs) without requiring specialized sparse execution kernels. The removal is performed along the output channel dimension of a convolutional layer.

Pruning a filter provides a direct and predictable reduction in computational cost and parameter count.

• FLOPs Reduction: Removing a filter eliminates all computations associated with its convolutional kernel across all input positions and input channels.
• Memory Reduction: The filter's weights are permanently removed, shrinking the model file size. This also reduces the memory footprint of intermediate feature maps in subsequent layers, as the number of output channels is reduced.

A pruning criterion is used to rank filters for removal. Common heuristics include:

• Magnitude-based: Filters with the smallest L1 or L2 norm of their weights are pruned first.
• Activation-based: Filters that produce feature maps with low average activation (e.g., near-zero) are considered less important.
• Gradient-based: Techniques like movement pruning score filters based on how much their weights change during fine-tuning. The scoring can be applied locally (per-layer) or globally (across the entire network).

To mitigate accuracy loss, filter pruning is typically not applied all at once. A common workflow is Iterative Magnitude Pruning (IMP):

1. Train a dense model to convergence.
2. Score and prune a small percentage (e.g., 10-20%) of the least important filters.
3. Fine-tune the pruned model to recover accuracy.
4. Repeat steps 2-3 until the target sparsity or performance threshold is met. This gradual process allows the network to adapt to the reduced capacity.

Filter pruning is inherently hardware-aware because its structured output aligns with the dense matrix-multiplication units in standard accelerators. Engineers can co-design the pruning strategy with the target hardware's characteristics, such as:

• Optimizing for the tiling dimensions of a matrix unit.
• Ensuring pruned layer dimensions are multiples of hardware-friendly values (e.g., 8 or 16 for vectorized instructions).
• Balancing reduction across layers to avoid creating bottlenecks that underutilize parallel compute resources.

The core challenge is managing the sparsity-accuracy tradeoff. Aggressive filter pruning increases compression and speed but risks significant accuracy degradation. The sensitivity to pruning varies greatly across layers; early and late convolutional layers are often more critical than middle layers. Analysis involves:

• Profiling the accuracy drop at different pruning rates per layer.
• Using regularization techniques like L1 regularization on filter norms during training to encourage more prune-friendly representations.
• The goal is to find the optimal per-layer sparsity profile that maximizes compression while staying within an acceptable accuracy budget.

The most common heuristic, which assumes filters with smaller weight norms contribute less to the output. The L1 norm (sum of absolute values) or L2 norm (sum of squared values) of a filter's kernel weights is calculated.

• Process: Compute the norm for each filter, rank them, and prune the smallest.
• Rationale: Small magnitude weights have a diminished effect during the forward pass.
• Advantage: Simple, computationally cheap, and requires no additional data.
• Limitation: A small-norm filter may still be critical if it activates for rare but important features.

Measures filter importance based on the sparsity of its output activations. A filter that frequently outputs zeros (inactive) across a dataset is considered less important.

• Process: Pass a calibration dataset, record the percentage of zero activations for each channel, and prune filters with the highest APoZ.
• Rationale: Filters that are rarely activated are redundant.
• Advantage: Data-driven; captures filter utility relative to actual input data.
• Limitation: Requires a representative dataset and forward passes to compute. Can be sensitive to the chosen activation threshold for 'zero'.

Estimates the change in the loss function if a specific filter were pruned, using a first-order Taylor expansion. The criterion approximates the saliency or importance of a parameter.

• Formula: Saliency ≈ |g * w|, where g is the gradient and w is the weight.
• Process: Compute gradients on a batch of data, calculate the saliency score for each filter, and prune those with the lowest scores.
• Rationale: Directly estimates the filter's contribution to the task loss.
• Advantage: More theoretically grounded than magnitude; connects pruning to training objective.
• Limitation: More expensive; requires gradient computation. Sensitive to gradient noise and batch selection.

Evaluates filter importance by analyzing the rank or redundancy of the feature maps they produce. Filters that generate linearly dependent or low-variance feature maps are pruned.

• Process: Perform Principal Component Analysis (PCA) on the stacked feature maps from a layer across a dataset. Filters contributing least to the principal components (e.g., with smallest eigenvalues) are pruned.
• Rationale: Aims to preserve the diversity and information content of the layer's output.
• Advantage: Reduces feature redundancy explicitly.
• Limitation: Computationally intensive for large feature maps; requires data for covariance estimation.

Exploits the scaling parameters in Batch Normalization (BN) layers that follow convolutional layers. In many architectures, a BN layer's scale parameter γ directly modulates the amplitude of each channel.

• Process: For channels with a small γ value (close to zero), the entire channel output is suppressed. Prune the corresponding convolutional filter that feeds into that channel.
• Rationale: The network, through training, has already learned to de-emphasize these channels.
• Advantage: Extremely efficient; the importance scores (γ) are a direct byproduct of training. Enables straightforward global pruning across layers.
• Limitation: Only applicable to architectures with specific BN-Conv patterns.

Selects filters not only for minimal accuracy loss but also for maximal inference speedup on target hardware. This may involve co-designing the pruning criterion with hardware performance models.

• Examples:
  • Latency-Aware: Uses a hardware latency lookup table to estimate speedup from pruning each filter; prunes filters that give the best latency reduction per unit of accuracy loss.
  • Energy-Aware: Models energy consumption of different layer configurations; prioritizes pruning filters in energy-intensive layers.
  • Pattern-Aware: Enforces pruning patterns (e.g., pruning entire filter groups) that align with hardware vector unit widths or memory burst sizes.
• Advantage: Maximizes real-world deployment efficiency, moving beyond FLOPs or parameter count as the sole metric.
• Challenge: Requires profiling the target hardware, making the criterion system-specific.

Structured pruning is a model compression technique that removes entire, regular structural components—such as neurons, channels, or filters—to maintain hardware-friendly, dense matrix operations. Unlike unstructured pruning, it produces a smaller, dense model that can leverage standard, optimized linear algebra libraries without requiring specialized sparse kernels.

• Key Benefit: Enables immediate speed-ups on general-purpose CPUs, GPUs, and NPUs.
• Common Targets: Removing entire filters (filter pruning), output channels (channel pruning), or attention heads in transformers.
• Trade-off: While more hardware-efficient, it is typically less fine-grained than unstructured pruning and may lead to a greater accuracy drop for the same parameter count reduction.

Channel pruning is a specific, widely-used form of structured pruning that removes entire output channels from a convolutional layer. This directly reduces the width of the output feature map and, critically, the number of input channels for the next convolutional layer, creating a cascading reduction in FLOPs and memory.

• Mechanism: Prunes based on the importance of an output channel, often measured by the L1 norm of its associated filter weights or its activation sparsity.
• Hardware Impact: Results in a dense network with fewer parameters and operations, leading to predictable latency improvements on standard hardware.
• Relation to Filter Pruning: In convolutional networks, channel pruning and filter pruning are often synonymous, as removing a filter is equivalent to removing its corresponding output channel.

N:M sparsity is a semi-structured sparsity pattern where, in every block of M consecutive weights (typically within a single row or channel of a tensor), at most N are non-zero. This pattern bridges the efficiency of structured pruning with the granularity of unstructured pruning.

• Hardware Advantage: Designed for and efficiently executed by sparse tensor cores in modern NVIDIA GPUs (e.g., Ampere architecture and later), providing up to 2x speedup for matrix operations.
• Pattern Example: A common pattern is 2:4 sparsity, where 50% of weights are pruned, but exactly two non-zero values remain in every block of four.
• Application: Often applied via fine-grained pruning algorithms that enforce this constraint, offering a favorable balance between high compression rates and hardware acceleration.

Hardware-aware pruning is a model compression approach that co-designs the sparsity pattern with the target hardware's execution engine and memory hierarchy. The goal is to maximize actual latency reduction or energy savings, not just theoretical parameter count reduction.

• Key Principle: The "best" sparsity pattern depends on the hardware. For a CPU, structured pruning is ideal. For a GPU with sparse tensor cores, N:M patterns are optimal.
• Process: Involves profiling the target hardware to understand its sensitivity to different data layouts, memory bandwidth constraints, and supported operations.
• Outcome: Produces a model whose sparsity is directly executable and efficient on the deployment platform, moving beyond abstract compression ratios.

A pruning criterion is the metric or heuristic used to score and rank parameters (or structural groups like filters) for removal. The choice of criterion fundamentally drives which parts of the network are considered redundant.

• Magnitude-Based: The most common criterion. Removes weights/filters with the smallest L1 or L2 norm, assuming low magnitude equates to low importance.
• Gradient-Based: Methods like Movement Pruning remove parameters based on how much their value changes during training, tying importance to the training process.
• Activation-Based: Scores filters based on the average percentage of zero activations they produce, pruning those that trigger less frequently.
• Second-Order: Advanced methods like Optimal Brain Damage (OBD) use the Hessian matrix to estimate a parameter's saliency to the loss function.

Sparse model inference refers to the runtime systems, execution kernels, and hardware support required to efficiently execute a pruned neural network where a significant portion of weights are zero. Simply having a sparse model does not guarantee faster inference; it requires a supporting software/hardware stack.

• Core Challenge: Skipping computations involving zero values to save FLOPs and memory bandwidth.
• Execution Methods:
  • Library Support: Using frameworks like PyTorch with torch.sparse or NVIDIA's cuSPARSELt for N:M sparsity.
  • Custom Kernels: Writing specialized CUDA or assembly kernels that leverage specific sparsity patterns.
  • Hardware Acceleration: Utilizing dedicated sparse compute units in modern NPUs and GPUs.
• Critical Consideration: The overhead of indexing and managing sparse data structures can negate benefits if sparsity is not high enough or not hardware-aligned.