Pruning

Pruning's primary outcome is model sparsity—the introduction of zeros into the network's weight matrices. The degree of sparsity is a key metric, often expressed as a percentage (e.g., 90% sparsity means 90% of weights are zero). This sparsity reduces:

• Memory footprint: Sparse matrices require less storage.
• Theoretical FLOPs: Zero-valued weights eliminate multiply-accumulate operations.
• Energy consumption: Fewer computations directly lower power draw, a critical factor for TinyML deployment on microcontrollers.

Pruning is categorized by the granularity of the elements it removes.

• Unstructured Pruning: Removes individual weights based on criteria like magnitude. Creates an irregular, sparse pattern that requires specialized software libraries or hardware (e.g., sparse tensor cores) for efficient execution.
• Structured Pruning: Removes entire, structurally regular components like neurons, channels, filters, or layers. Produces a smaller, dense network that runs efficiently on standard hardware without specialized runtimes. N:M Sparsity (e.g., 2:4) is a fine-grained structured pattern where for every block of M weights, N are zero, supported by modern accelerators.

The algorithm for selecting which parameters to prune. Common criteria include:

• Magnitude-based: Prunes weights with the smallest absolute values (L1 norm), a simple and effective baseline.
• Gradient-based: Uses gradient information to estimate a parameter's importance to the loss function.
• Hessian-based: More computationally expensive methods that estimate the impact on loss using second-order derivatives.
• Activation-based: Prunes neurons or channels that contribute minimally to the next layer's activation. The choice of criterion directly impacts the final accuracy and the recoverability of the pruned model.

Pruning is rarely a one-shot operation. The standard methodology is iterative pruning:

1. Train a dense model to convergence.
2. Prune a small percentage (e.g., 20%) of parameters based on the chosen criterion.
3. Fine-tune the remaining network to recover lost accuracy.
4. Repeat steps 2-3 until the target sparsity or performance threshold is met. This gradual approach, coupled with fine-tuning, is essential to mitigate the accuracy drop from aggressive pruning. It aligns with findings related to the Lottery Ticket Hypothesis.

The practical benefits of pruning are contingent on deployment infrastructure.

• Unstructured sparsity requires sparse linear algebra libraries (e.g., cuSPARSE) or dedicated hardware support to skip zero operations and realize speedups.
• Structured sparsity yields immediately deployable, smaller models compatible with all dense hardware accelerators.
• Compiler optimization: Frameworks like TensorFlow Lite for Microcontrollers and Apache TVM can leverage pruning-induced sparsity to generate optimized code for microcontrollers, translating sparsity into actual latency and energy savings.

Pruning is most powerful when combined with other model compression techniques in a pipeline:

• Pruning then Quantization: A pruned model is often more robust to the precision loss from post-training quantization (PTQ) or quantization-aware training (QAT), as there are fewer parameters to quantize.
• Pruning with Knowledge Distillation: A pruned model can serve as the student in distillation, learning from a larger teacher to regain accuracy.
• Pruning within NAS: Hardware-aware neural architecture search can use pruning metrics as constraints to discover inherently efficient architectures. This combinatorial approach is standard for extreme TinyML deployment.

Unstructured pruning removes individual weights based on a criterion like magnitude, creating an irregular, sparse pattern. This method offers high theoretical compression but requires specialized software or hardware (like sparse tensor cores) for efficient execution, as standard dense matrix multiplication cannot leverage the sparsity.

• Criteria: Typically uses weight magnitude (L1 norm) or gradient-based saliency scores.
• Result: A highly sparse weight matrix (e.g., 90% zeros).
• Challenge: The irregular memory access pattern often negates speed benefits on standard MCUs without dedicated sparse kernels.

Structured pruning removes entire, structurally regular components like neurons, channels, filters, or layers. This produces a smaller, denser network architecture that is immediately executable on standard hardware without specialized libraries, making it the preferred method for microcontroller deployment.

• Common Targets: Pruning entire convolutional filters, attention heads in transformers, or neurons in fully-connected layers.
• Hardware-Friendly: Results in a cleanly smaller model that directly reduces FLOPs and memory footprint.
• Trade-off: Often leads to greater accuracy loss for the same parameter reduction compared to unstructured pruning, as it is less granular.

This is the most common practical algorithm for applying pruning. Instead of pruning once, it follows a cycle: train → prune the smallest-magnitude weights → fine-tune. This process repeats over multiple iterations, allowing the network to gradually adapt to the sparsity.

• Process: A target sparsity (e.g., 50%) is achieved over multiple pruning steps (e.g., 20% per step).
• Benefit: Significantly preserves accuracy compared to one-shot pruning.
• Foundation: Empirical validation for many pruning techniques, providing a stable baseline for comparison.

A hybrid approach that imposes a regular, hardware-efficient sparsity pattern. For every block of M weights (e.g., 4), at least N (e.g., 2) must be zero. This pattern is efficiently supported by modern NVIDIA Ampere/Hopper GPU tensor cores for acceleration.

• Pattern: Example: 2:4 sparsity, meaning 50% of weights are pruned in a structured, block-wise manner.
• Hardware Support: Enables speedups on supported accelerators without custom sparse kernels.
• Application: While initially for GPUs, research explores applying similar block-sparse patterns for efficient CPU/MCU inference.

A influential conjecture stating that a dense, randomly-initialized network contains a subnetwork (a 'winning ticket') that, when trained in isolation, can match the accuracy of the full network. This motivates pruning at initialization.

• Implication: Ideal pruning should identify this trainable subnetwork early.
• Algorithm: Iterative Magnitude Pruning with rewinding (resetting weights to early training values) often finds strong tickets.
• Impact: Drives research into identifying sparse, trainable architectures from the start of training.

For microcontroller deployment, structured pruning is typically the first choice due to its compatibility with standard inference engines. The workflow integrates with other compression techniques:

1. Train a dense model to a good accuracy baseline.
2. Apply iterative structured pruning (e.g., channel pruning) followed by fine-tuning.
3. Quantize the resulting smaller, pruned model using Post-Training Quantization or Quantization-Aware Training.
4. Compile the final pruned-and-quantized model for the target MCU (e.g., using TensorFlow Lite for Microcontrollers).

This combined approach maximizes the reduction in model size, RAM usage, and inference latency.

Pruning is categorized by the pattern of parameters it removes.

• Structured Pruning: Removes entire structural components like neurons, channels, or filters. This results in a smaller, dense network that runs efficiently on standard hardware.
• Unstructured Pruning: Removes individual weights based on criteria like magnitude. This creates an irregular, sparse model that can achieve high compression ratios but requires specialized software or hardware (e.g., sparse matrix libraries) for speedup.
• Trade-off: Structured pruning offers easier deployment; unstructured pruning offers higher potential compression.

These are parameter reduction techniques that complement pruning.

• Weight Clustering (or Weight Sharing): Groups similar weight values into clusters. Each weight is then stored as a small index into a shared codebook of centroid values, dramatically reducing storage.
• Low-Rank Factorization: Approximates a large weight matrix (e.g., in a fully-connected layer) as the product of two or more smaller matrices. This reduces the total number of parameters and the computational cost of the layer operation.
• Application: Effective for compressing models where pruning alone is insufficient, often applied to fully-connected layers.

Pruning creates model sparsity—a high percentage of zero-valued weights. Exploiting this sparsity is key to achieving actual speed and energy gains.

• Structured Sparsity (e.g., N:M): Patterns like 2:4 sparsity (2 zeros in every block of 4 weights) are directly supported by modern GPU/TPU tensor cores for acceleration.
• Sparse Kernels: Specialized software libraries execute sparse matrix multiplications, skipping operations involving zeros.
• Hardware Support: Emerging AI accelerators and some microcontrollers include features to efficiently skip zero weights, making pruned models not just smaller but faster.