Pruning (Neural Network)

Unstructured pruning removes individual weights based on a saliency criterion (e.g., magnitude), creating an irregular, sparse network. It offers high compression ratios but requires specialized software or hardware to exploit sparsity for speed gains.

• Method: Iteratively zero out weights with the smallest absolute values.
• Challenge: The resulting sparse matrix pattern is irregular, limiting acceleration on standard hardware without dedicated sparse linear algebra kernels.
• Use Case: Foundational research and extreme compression where model size, not inference latency, is the primary constraint.

Structured pruning removes entire structural components like neurons, channels, filters, or layers. This results in a smaller, dense network that runs efficiently on standard hardware.

• Methods: Prune based on the L2 norm of filters or the activation importance of channels.
• Advantage: Produces directly executable, smaller models without requiring sparse computation libraries.
• Trade-off: Typically achieves lower compression rates than unstructured pruning for a given accuracy drop, as it is a more constrained optimization problem.

A seminal algorithm for unstructured pruning. It follows a train, prune, fine-tune cycle repeated over multiple iterations.

• Process:
  1. Train a dense model to convergence.
  2. Remove a percentage (e.g., 20%) of the smallest-magnitude weights.
  3. Fine-tune the remaining weights to recover accuracy.
  4. Repeat steps 2-3 until target sparsity is reached.
• Origin: Popularized by the Lottery Ticket Hypothesis, which found that the sparse sub-networks ("winning tickets") exist at initialization within dense networks.

This distinction defines the scope for selecting which parameters to prune.

• Global Pruning: Ranks all weights in the model across all layers against a single criterion. It removes the globally least important weights, allowing more flexibility in resource allocation across layers.
• Local Pruning: Applies the pruning criterion independently within each layer or structural group (e.g., prune 30% of weights in each convolutional filter separately). It's simpler to implement and ensures no layer is completely removed, but may be less optimal than a global strategy.

These strategies define the aggressiveness of the pruning schedule.

• One-Shot Pruning: Removes the target percentage of weights in a single step after training, followed by a fine-tuning phase. It is computationally cheap but can lead to significant, irrecoverable accuracy loss at high sparsities.
• Gradual Pruning: Incrementally increases sparsity over many training steps (e.g., after each mini-batch update). This allows the network to adapt continuously to the changing architecture, typically preserving final accuracy much better than one-shot pruning, especially at high compression rates.

The metric used to determine a parameter's importance. The choice of criterion directly impacts which weights are removed.

• Magnitude (L1 Norm): The most common criterion. Assumes smaller absolute weights contribute less to the output.
• Gradient-based Saliency: Estimates the effect on the loss function if a weight is removed.
• Hessian-based (Optimal Brain Damage/Surgeon): Uses second-order derivative information to approximate the increase in error caused by removing a parameter. More accurate but computationally expensive.
• Activation-based: For structured pruning, uses the average magnitude of a channel's activations across a dataset as its importance score.

Quantization reduces the numerical precision of a model's weights and activations, typically from 32-bit floating-point (FP32) to lower formats like 16-bit (FP16), 8-bit integers (INT8), or even 4-bit (INT4). This directly shrinks the model's memory footprint and can accelerate inference on hardware that supports low-precision arithmetic.

• Post-Training Quantization (PTQ): Applied after training with minimal calibration data.
• Quantization-Aware Training (QAT): Simulates quantization during training for better accuracy preservation.
• Key Benefit: Enables larger models to fit into limited device memory (e.g., on mobile or edge devices).

Knowledge distillation is a model compression and transfer learning technique where a smaller, more efficient student model is trained to mimic the behavior of a larger, more accurate teacher model. The student learns not just from the original training data labels (hard targets) but from the teacher's softened output probabilities (soft targets), which contain richer information about class similarities.

• Process: The teacher's logits are used to create a softmax distribution with a temperature parameter, providing a 'dark knowledge' signal.
• Outcome: A compact model that often outperforms one trained solely on the original data.

Low-rank factorization approximates a large weight matrix W (of dimensions m x n) as the product of two or more smaller matrices (e.g., U (m x r) and V (r x n), where r << m, n). This exploits the idea that many weight matrices in trained neural networks have a low intrinsic rank, meaning their information can be represented more compactly.

• Mathematical Basis: Applies techniques like Singular Value Decomposition (SVD) to find an optimal low-rank approximation.
• Application: Particularly effective for compressing the large linear layers in transformers and CNNs.
• Result: Reduces the total parameter count from mn to r(m+n).

This distinction defines the pattern in which parameters are removed.

• Unstructured Pruning: Removes individual weights anywhere in the network, resulting in irregular, fine-grained sparsity. While it can achieve high compression rates, it requires specialized software or hardware libraries to realize speedups.
• Structured Pruning: Removes larger, contiguous structural components like entire neurons, channels, filters, or layers. This results in a smaller, dense network that can leverage standard hardware and software for immediate acceleration.

Trade-off: Structured pruning is more hardware-friendly but can be less precise than unstructured pruning.

Deep compression is a famous three-stage pipeline for extreme neural network compression, introduced by Han et al. It combines multiple techniques sequentially for multiplicative savings.

1. Pruning: Removes redundant connections.
2. Quantization: Reduces the precision of the remaining weights (e.g., weight sharing).
3. Huffman Coding: Applies lossless entropy coding to the quantized weights.

• Result: This pipeline achieved a 35x to 49x reduction in the size of AlexNet without loss of accuracy, enabling large vision models to fit in on-chip SRAM.

Sparse training aims to train a neural network from scratch with a fixed, sparse connectivity pattern, bypassing the traditional 'train large, then prune' paradigm. This avoids the computational cost of training a large dense model only to throw most of it away.

• Methods: Includes algorithms like RigL (Rigged Lottery) and SET (Sparse Evolutionary Training) that dynamically grow and prune connections during training.
• Advantage: Can lead to more efficient training cycles and models whose sparse structure is inherently optimized.
• Challenge: Requires careful management of the sparse topology to maintain trainability and final accuracy.