Structured Pruning

Structured pruning operates on predefined structural units within a neural network. The most common granularities are:

• Channel Pruning: Removes entire output channels from a convolutional layer's weight tensor, along with corresponding input channels in the subsequent layer.
• Filter Pruning: Removes entire 3D filters from a convolutional layer, effectively removing output channels.
• Neuron Pruning: In fully connected layers, removes entire neurons (output nodes). This results in a smaller, but still dense, weight matrix that can leverage standard matrix multiplication libraries.

The primary advantage over unstructured pruning is hardware compatibility. By removing entire structural groups, the resulting model:

• Uses dense linear algebra (GEMM) kernels, which are highly optimized on all hardware, including microcontroller NPUs.
• Avoids the need for sparse kernels and specialized hardware support, which are rare on MCUs.
• Provides a predictable, linear relationship between the pruning ratio and the reduction in FLOPs and memory footprint. Removing 30% of channels yields a ~30% reduction in compute for that layer.

Effective structured pruning is rarely a one-step process. It typically follows an iterative cycle to recover accuracy:

1. Train a baseline model to convergence.
2. Evaluate Importance of each structural group (e.g., using L1/L2 norm of filters, activation impact).
3. Remove the least important groups.
4. Fine-tune the smaller, pruned model to recover lost accuracy.
5. Repeat steps 2-4 until the target size or accuracy threshold is met. This cycle allows the model to adapt its remaining weights to compensate for the removed structures.

A key characteristic is the preservation of the model's fundamental layer-wise architecture. The pruned model is a strict subset of the original, with fewer channels/filters per layer. This means:

• No irregular memory access patterns that cripple performance on small MCU caches.
• Zero overhead for storing sparse matrix indices (unlike CSR/CSC formats in unstructured pruning).
• Simplified deployment using the same inference engine; only the weight tensors are smaller. The trade-off is potentially less theoretical compression than ideal unstructured pruning, but far superior real-world latency on MCUs.

Determining which channels or filters to prune requires a scoring function. Common metrics include:

• Magnitude-based: L1 or L2 norm of the filter weights. Small norm implies low importance.
• Activation-based: Average percentage of zeros in a channel's activations (using a ReLU). High sparsity implies low importance.
• Gradient-based: First-order Taylor expansion to estimate the impact on the loss function.
• Regularization: Adding L1 or group Lasso regularization to filter weights during training to encourage sparsity in structures. The choice of metric balances computational cost and final model accuracy.

Understanding structured pruning is best done by contrasting it with its counterpart:

• Unstructured Pruning: Removes individual weights anywhere in the network. Results in highly sparse matrices (e.g., 90% zeros).
• Theoretical vs. Practical Compression: Unstructured can achieve higher element-wise sparsity, but requires sparse libraries for speedup, which are inefficient on MCUs without dedicated hardware.
• Structured Pruning: Removes blocks of weights. Results in smaller dense matrices. Delivers reliable, readily achievable speedups on commodity microcontroller cores (Cortex-M) and NPUs. For MCU deployment, structured pruning is often the default choice due to its deterministic performance benefits.

Pruning entire channels (also called filters) from a convolutional layer's output. This is the most common form of structured pruning. Removing a channel in layer L allows for the removal of the corresponding kernel in all filters of the subsequent layer L+1, leading to significant reductions in both parameters and FLOPs. The resulting model has fewer output channels, making it directly executable by standard dense linear algebra libraries without sparse kernels.

Pruning entire attention heads within a transformer's multi-head attention (MHA) block. Each head operates on a separate, learned subspace. Research shows many heads are redundant. Pruning them reduces the projection matrices for queries, keys, and values, shrinking the model's parameter count and the computational cost of the attention operation. This is critical for deploying small language models on edge devices.

Pruning entire neurons (or units) in fully connected (dense) layers. This involves removing an output neuron and all its incoming weights. It is analogous to channel pruning for convolutional layers. The benefit is a dense, smaller weight matrix for the layer. This target is highly effective for models with large MLP blocks, such as vision transformers (ViTs) and multilayer perceptrons.

Pruning entire layers or residual blocks from a deep network. This is a coarse-grained approach that directly reduces network depth. It is often applied to over-parameterized models (e.g., large ResNets) where certain layers contribute minimally to the final output. The challenge is maintaining the data flow; in residual networks, the skip connection must be preserved or adapted when an intermediate block is removed.

Pruning based on stride or dilation patterns within convolutional kernels. For example, removing every other weight in a kernel according to a fixed pattern. While this creates sparsity, the pattern is regular and predictable, allowing for custom, efficient kernels that skip the pruned positions. This approach sits between unstructured and fully structured pruning, offering a balance of flexibility and hardware efficiency.

Pruning entire groups within a grouped convolution. In a grouped convolution (e.g., as used in ResNeXt or MobileNet), channels are partitioned into independent groups. Pruning entire groups reduces the width of the layer uniformly across groups. This is a natural structural target because the groups are designed to be computationally independent, making removal clean and efficient.

Model pruning is a broader model compression technique that removes redundant or less important parameters (weights) from a neural network to reduce its size and computational cost. Unlike structured pruning, it often results in unstructured sparsity, where individual weights are set to zero, creating irregular, sparse weight matrices. Exploiting this sparsity typically requires specialized sparse kernels or hardware.

Sparsity refers to the proportion of zero-valued elements in a tensor, particularly a weight matrix. It is the direct outcome of pruning techniques.

• Unstructured Sparsity: Created by standard pruning; zeros are randomly distributed. Hard to accelerate on standard hardware.
• Structured Sparsity: Created by structured pruning; entire rows, columns, or blocks of weights are zero, resulting in a smaller, dense sub-network that is easily accelerated.

Quantization is a complementary model compression technique that reduces the numerical precision of a neural network's weights and activations (e.g., from 32-bit floating-point to 8-bit integers). This decreases model size and enables the use of efficient integer arithmetic units on microcontrollers. It is often applied after structured pruning to further compress the resulting dense model.

• Key methods: Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT).

Depthwise separable convolution is an efficient neural network layer design that is inherently smaller and less compute-intensive than standard convolutions. It is a prime target for structured pruning. The layer works in two steps:

1. Depthwise Convolution: Applies a single filter per input channel.
2. Pointwise Convolution (1x1 convolution): Combines the outputs across channels. Pruning entire filters from this structure directly reduces feature maps and parameters.

Hardware-Aware Neural Architecture Search (NAS) is an automated method for discovering optimal neural network designs given specific hardware constraints (e.g., latency, memory). It can be used to co-design efficient architectures that are inherently more amenable to techniques like structured pruning. NAS algorithms evaluate thousands of candidate architectures, often finding novel, highly efficient building blocks for TinyML.

Kernel optimization is the low-level tuning of fundamental neural network operation implementations for a specific microcontroller CPU. After structured pruning creates a smaller, dense model, these hand-optimized kernels (e.g., for matrix multiplication or convolution) maximize performance. Common techniques include:

• Loop Unrolling & Tiling
• SIMD Instruction usage
• Operator Fusion (e.g., fuse Conv, BatchNorm, and Activation) Frameworks like CMSIS-NN provide libraries of such optimized kernels for Arm Cortex-M processors.