Model Pruning

Magnitude-based pruning is the most common unstructured pruning technique. It operates on the principle that weights with small absolute values contribute less to the model's output. The process is iterative:

• Weights below a chosen threshold are set to zero.
• The remaining non-zero weights are often fine-tuned to recover accuracy.
• This creates an irregular, sparse weight matrix.

Key Consideration: While highly effective at reducing parameter count, the resulting irregular sparsity often requires specialized sparse linear algebra libraries (not commonly available on MCUs) or dedicated hardware to realize computational speedups.

Structured pruning removes entire structural components of a neural network, such as filters, channels, or layers, resulting in a physically smaller, dense model. This is in contrast to the irregular sparsity of unstructured pruning.

Common Targets:

• Filter Pruning: Removing entire convolutional filters.
• Channel Pruning: Removing channels from feature maps.
• Layer Pruning: Excising entire layers from the network.

Advantage for TinyML: The output is a smaller, dense network that can leverage standard, optimized dense linear algebra kernels (like CMSIS-NN) on microcontrollers without requiring sparse compute support, leading to predictable latency and memory reductions.

Iterative pruning (or iterative magnitude pruning) is a process that alternates between pruning a small percentage of low-magnitude weights and fine-tuning the network. This gradual approach is superior to one-shot pruning for preserving model accuracy.

Typical Workflow:

1. Train a large model to convergence.
2. Prune a small fraction (e.g., 20%) of the smallest-magnitude weights.
3. Fine-tune the pruned model to recover accuracy.
4. Repeat steps 2-3 until the target sparsity is met.

This method allows the network to adapt to the changing architecture, often achieving higher sparsity levels with less accuracy degradation than aggressive one-shot pruning.

The Lottery Ticket Hypothesis is a theoretical framework that informs pruning strategy. It posits that within a large, dense neural network, there exists a smaller winning ticket subnetwork that, if trained in isolation from the start, can match the accuracy of the original network.

Implication for Pruning: Pruning is not just about removing unimportant weights, but about identifying this optimal sparse architecture. The Iterative Magnitude Pruning process is a practical method for discovering these tickets. Foundational papers, such as "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks" by Frankle & Carbin, provide the empirical basis for this approach.

Hardware-aware pruning integrates direct feedback from the target deployment hardware into the pruning loop. The goal is to produce a model that is not just sparse, but also optimized for specific hardware metrics like latency, energy, or memory bandwidth.

How it works:

1. A candidate model is pruned.
2. Its latency or energy consumption is profiled on the target device (or an accurate simulator/emulator).
3. This hardware performance metric is used as a reward or constraint to guide the pruning algorithm.

This method moves beyond abstract sparsity to produce models with predictable, improved on-device performance, which is critical for TinyML system design.

Also known as dynamic sparse training, this method integrates pruning into the initial training phase rather than applying it as a post-training step. The network is trained with sparsity enforced from the beginning.

Mechanisms include:

• Gradual Pruning Schedules: The sparsity level is increased according to a schedule during training.
• Sparse Regularization: Techniques like L0 regularization are used to encourage weights to become exactly zero.
• Weight Rewinding: Weights are periodically reset to an earlier point in training while maintaining the sparse mask.

Benefit: This can be more computationally efficient than the traditional train-prune-fine-tune pipeline and can help discover better sparse architectures by co-optimizing weight values and connectivity.

A compression technique that reduces the numerical precision of a model's weights and activations (e.g., from 32-bit floating-point to 8-bit integers). This drastically cuts the model's memory footprint and enables the use of efficient integer arithmetic on hardware without floating-point units.

• Primary Benefit: Reduces model size and accelerates inference.
• Common Types: Post-training quantization (PTQ) and quantization-aware training (QAT).
• Hardware Target: Essential for deployment on microcontrollers (MCUs) and digital signal processors (DSPs).

A training process where a compact student model learns to replicate the behavior of a larger, more accurate teacher model. The student is trained not just on raw data labels, but to match the teacher's softened output probabilities (logits) or intermediate feature representations.

• Core Mechanism: Transfers 'dark knowledge' from a high-capacity model to a smaller one.
• TinyML Application: Creates highly accurate small models suitable for edge deployment where the teacher model would be too large.
• Outcome: Often achieves better accuracy than a small model trained directly on the data.

An automated process for designing optimal neural network architectures. In the context of edge AI, Hardware-Aware NAS (HW-NAS) directly incorporates constraints like latency, memory usage, and energy consumption into the search objective.

• Goal: Discovers model architectures that are Pareto-optimal for a given hardware target.
• Example Framework: MCUNet uses TinyNAS to co-design models with its inference engine.
• Result: Architectures that are inherently efficient, not just compressed after the fact.

A compiler-level optimization that merges multiple sequential neural network operations into a single, compound kernel. This is critical for reducing memory accesses and eliminating the need to store large intermediate tensors in scarce RAM.

• How it Works: For example, a Convolution, BatchNorm, and Activation (ReLU) layer can be fused into one operation.
• Benefit: Significantly reduces inference latency and peak memory usage on edge devices.
• Implementation: Performed by inference engines like TFLite Micro and TinyEngine (part of MCUNet).

The total amount of memory (both RAM and Flash/ROM) required to store and execute a machine learning model on a target device. It is the ultimate constraint for TinyML deployment.

• Flash: Stores the model's weights and the application code.
• RAM (Working Memory): Holds the model's activations, input/output buffers, and intermediate tensors during inference.
• Optimization Goal: Pruning, quantization, and fusion all directly aim to minimize this footprint to fit within the severe limits of microcontrollers (often < 512KB RAM, < 1MB Flash).