Model Pruning

Unstructured pruning removes individual weights based on a saliency criterion (e.g., magnitude), creating an irregular, sparse weight matrix. This method can achieve high theoretical sparsity but requires specialized software or hardware to exploit the sparsity for speed gains.

• Criteria: Often uses magnitude-based pruning (smallest weights).
• Result: Highly irregular sparsity pattern.
• Challenge: Requires sparse linear algebra libraries or dedicated hardware (like sparse tensor cores) for acceleration. On standard microcontrollers, unstructured sparsity often provides memory savings but not guaranteed latency reduction.

Structured pruning removes entire structural components, such as filters, channels, or neurons, resulting in a smaller, denser network. This is hardware-friendly as it maintains regular data layouts compatible with standard optimized kernels.

• Granularity: Prunes filters (Conv2D), attention heads (Transformers), or neurons (Dense layers).
• Result: A smaller, architecturally coherent model.
• Advantage: Directly reduces FLOPs and memory footprint. The pruned model can be accelerated with standard dense linear algebra libraries like CMSIS-NN or TensorFlow Lite for Microcontrollers without specialized sparse support.

Magnitude-based pruning is the most common criterion, where weights with the smallest absolute values are considered least important and removed. It's simple, computationally cheap, and often effective.

• Process: After training, weights below a threshold are set to zero.
• Iterative Pruning: Often applied iteratively: prune a small percentage of weights, fine-tune the network to recover accuracy, and repeat.
• Use Case: Forms the basis for many pruning algorithms. For microcontroller deployment, magnitude-based structured pruning of entire channels is common to yield directly executable smaller models.

This is a process, not a single criterion. Iterative pruning involves cycling between removing parameters and fine-tuning the remaining network to recover lost accuracy, allowing for more aggressive compression.

• Typical Cycle: Prune 10-20% of parameters → Fine-tune for a few epochs → Repeat.
• Benefit: Achieves higher final compression rates with less accuracy degradation compared to one-shot pruning.
• TinyML Consideration: The fine-tuning stage requires significant compute and a training dataset, which may be done offline before deploying the final, static pruned model to the microcontroller.

Inspired by the Lottery Ticket Hypothesis, this method identifies and trains a sparse subnetwork from the start. A network is initially pruned, and the remaining weights are reset to their original initialization values before being retrained.

• Finding: Such 'winning ticket' subnetworks can often match the accuracy of the original dense network.
• Implication: Suggests that efficient, trainable subnetworks exist within larger over-parameterized models.
• Application: Can be used to find highly sparse models suitable for deployment, though the search process itself can be computationally intensive.

This defines the scope for applying the pruning threshold.

• Layer-Wise Pruning: Applies a threshold independently to each layer. A common approach that's simple but may prune important layers too aggressively or unimportant layers too lightly.
• Global Pruning: Applies a single threshold across all weights in the network. This automatically allocates sparsity to less sensitive layers, often yielding better accuracy for a given overall sparsity level. It requires collecting and sorting all network weights, which has a higher memory overhead during the pruning process.

Pruning directly shrinks the model size stored in non-volatile flash memory. On microcontrollers with 256KB-1MB of flash, this is often the primary deployment constraint.

• Structured pruning removes entire filters or channels, producing a smaller, dense model that fits directly into flash.
• Unstructured pruning creates sparse weight matrices. Specialized formats like Compressed Sparse Row (CSR) or block sparsity are used to store only non-zero values and their indices, achieving compression ratios of 2x-10x.
• This reduction is essential for over-the-air (OTA) updates, minimizing transmission time and cost for device fleets.

Peak RAM footprint during inference is dominated by activation maps. Pruning reduces the number of computations, which in turn reduces the size of intermediate tensors.

• Fewer parameters mean smaller kernel sizes and output channels, leading to smaller activation buffers.
• This enables in-place computation and static memory allocation strategies, where all buffers are pre-allocated at compile-time for deterministic execution.
• Reduced memory traffic lowers power consumption, a critical factor for battery-powered IoT sensors.

By zeroing out weights, pruning eliminates multiply-accumulate (MAC) operations. On MCUs without dedicated AI accelerators, this directly translates to lower latency and CPU cycles.

• Structured pruning (e.g., removing a 3x3 convolutional filter) eliminates all associated MACs, offering predictable speedups on standard DSP or CMSIS-NN kernels.
• Unstructured sparsity requires specialized sparse kernels to skip zero-operand multiplications. Frameworks like TensorFlow Lite Micro can leverage this for up to 2-3x latency improvements on compatible cores.
• This allows for higher inference frequencies or frees the CPU for other critical tasks.

Pruning and quantization are synergistic techniques. A pruned, simpler model often quantizes to lower bit-widths (e.g., INT4) with less accuracy degradation.

• Removing outlier weights through pruning results in a tighter, more uniform weight distribution. This simplifies the calibration process for post-training quantization (PTQ).
• The combined pipeline of pruning -> fine-tuning -> quantization-aware training (QAT) is standard for producing ultra-low-bit models (e.g., pruned INT4) that are viable for microcontroller deployment.
• This combination is key for deploying larger model architectures, like MobileNetV2, on Cortex-M4/M7 class devices.

Effective pruning for MCUs must account for the target hardware's capabilities. Naive pruning can hinder performance if not aligned with hardware optimizations.

• Filter pruning aligns well with SIMD instructions (e.g., ARM's SMLAD), as it removes entire vectors of computation.
• Pruning patterns can be designed to leverage operator fusion opportunities in the compute graph, such as fusing a pruned convolution with a subsequent activation.
• For devices with tiny scratchpad memories, pruning is used to reduce tensor dimensions to fit within SRAM blocks, avoiding costly external memory access.

Pruning is integrated into the TinyML deployment pipeline via frameworks and compilers that handle sparse model formats and code generation.

• Frameworks like TensorFlow Lite Micro and PyTorch Mobile support importing models pruned with libraries like TensorFlow Model Optimization Toolkit.
• The final deployment involves a compiler (e.g., TVM, Apache TVM Micro) that can translate the pruned model graph into optimized C code with static scheduling.
• The toolchain must map sparse operations to efficient kernel implementations, often using hand-optimized libraries like CMSIS-NN for Arm Cortex-M cores.

Quantization reduces the numerical precision of a model's weights and activations, typically from 32-bit floating-point (FP32) to lower-bit integers like 8-bit (INT8). This directly shrinks the model's flash footprint and enables the use of efficient integer arithmetic units common in microcontrollers.

• Post-Training Quantization (PTQ): Converts a pre-trained model using a calibration dataset, requiring no retraining.
• Quantization-Aware Training (QAT): Trains the model with simulated quantization, often yielding higher accuracy than PTQ.
• The primary trade-off is a potential, managed loss in model accuracy for massive gains in efficiency.

Structured pruning removes entire, coherent structures from a neural network, such as entire filters in a convolutional layer or neurons in a dense layer. This results in a smaller, densely connected model that maintains a standard shape and can be accelerated with conventional, optimized kernels without requiring specialized sparse hardware support.

• Contrasts with unstructured pruning, which creates irregular sparsity.
• Directly reduces both the number of parameters and the computational FLOPs (floating-point operations).
• The pruned model is often retrained (fine-tuned) to recover accuracy.

Knowledge distillation is a training technique where a small, efficient model (the student) is trained to mimic the behavior of a larger, more accurate model (the teacher). The student learns not just from the hard data labels, but from the teacher's soft labels (probability distributions), which contain richer information about class similarities.

• Enables the creation of highly compact models that retain much of the performance of larger architectures.
• Often used in conjunction with pruning and quantization as a final training step.
• Particularly effective for creating Tiny Language Models and other complex, compact networks.

Sparsity is a measure of the proportion of zero-valued elements in a tensor, most commonly a weight matrix. Pruning intentionally induces high sparsity. Sparse models can skip computations involving zeros, but realizing this speedup on microcontrollers requires specialized techniques.

• Unstructured Sparsity: Zeros are randomly distributed; exploiting it requires sparse matrix libraries which may have overhead on MCUs.
• Structured Sparsity: Zeros appear in blocks (e.g., entire channels), making acceleration more straightforward.
• The sparsity ratio is a key metric: (number of zero parameters / total parameters).

Neural Architecture Search automates the design of neural network architectures. For TinyML, Hardware-Aware NAS directly incorporates target device constraints (e.g., latency, memory, power) into the search objective. This can discover inherently efficient architectures that require less aggressive post-training pruning.

• Searches over operations (e.g., depthwise separable convolutions), layer counts, and channel widths.
• Can produce Pareto-optimal models that balance accuracy and resource use.
• Complements pruning by designing networks where unimportant components are minimal from the start.

Weight clustering, or weight sharing, is a compression technique that groups similar weight values into a smaller number of shared centroids. The model stores only the centroid indices and a small codebook of centroid values, dramatically reducing the storage bits per weight.

• For example, a layer with 1,000 weights might be represented using only 16 unique centroid values (4-bit indices).
• Inference involves looking up centroid values, adding minimal computational overhead.
• Often applied after pruning and quantization for additional compression on the model footprint.