Weight Pruning

This is the most common unstructured pruning technique. It operates on the principle that weights with small absolute values contribute less to the model's output. The process is iterative:

• Pruning: Remove weights below a certain threshold.
• Fine-tuning: Retrain the network to recover accuracy lost from pruning.
• Repeat: Iterate the prune/fine-tune cycle to achieve the target sparsity. It's simple and effective but requires careful tuning of the sparsity schedule and threshold to avoid significant accuracy loss.

Unlike unstructured pruning, structured pruning removes entire structural components of the network, such as:

• Filters/Channels in convolutional layers.
• Neurons in fully-connected layers.
• Attention Heads in transformer blocks. This results in a smaller, denser model that is inherently more efficient on standard hardware (CPUs/GPUs) because it eliminates the need for sparse matrix computation libraries. The challenge is identifying which structures are least important, often using metrics like L1/L2 norm of filters or activation-based importance scores.

A best-practice methodology that prunes a network gradually over multiple cycles rather than in a single, aggressive step.

• Process: Prune a small percentage (e.g., 10-20%) of weights, then fine-tune the model to recover performance. Repeat.
• Benefit: Allows the network to adapt and re-distribute importance among remaining weights, preserving significantly higher accuracy compared to one-shot pruning to the same final sparsity.
• Scheduling: The pruning rate can be constant, gradual, or follow a schedule (e.g., cubic sparsity).

A influential research finding that identifies winning tickets within dense networks. The hypothesis states:

• A large, randomly-initialized network contains a subnetwork that, when trained in isolation, can match the accuracy of the original network.
• Process: Train a network, prune it, and reset the remaining weights to their original initial values (the "winning ticket"). This subnetwork can then be trained to full accuracy, often faster. This challenges the view that pruning merely compresses a trained model, suggesting it can also find more trainable, efficient architectures from the start.

The end-goal of pruning for edge deployment. Key considerations include:

• Hardware Compatibility: Unstructured sparsity often requires specialized libraries (e.g., DeepSparse) or hardware (certain NPUs) for speedups. Structured pruning yields models compatible with standard frameworks like TensorFlow Lite or PyTorch Mobile.
• Sparsity Pattern: Block sparsity (pruning groups of weights) can offer a better balance, providing some regularity for acceleration without the rigidity of full structured pruning.
• Toolchains: Frameworks like TensorFlow Model Optimization Toolkit and PyTorch's torch.nn.utils.prune provide APIs to integrate pruning into the training pipeline for eventual edge deployment.

Pruning is rarely used in isolation. It is part of a broader model compression stack, often combined with:

• Quantization: After pruning, the remaining weights are quantized (e.g., to INT8) for further memory and compute reduction. The order (prune then quantize) is typical.
• Knowledge Distillation: A pruned model can serve as the student, learning from the original dense teacher model to regain accuracy.
• Neural Architecture Search (NAS): Can be used to automatically discover pruning policies or architectures that are inherently sparse and efficient. This combination enables the extreme model size reductions required for TinyML and microcontroller deployment.

The most common unstructured pruning strategy, where weights with the smallest absolute magnitude are considered least important and set to zero. This is often implemented as an iterative process.

• Implementation: Frameworks like TensorFlow Model Optimization Toolkit and PyTorch's torch.nn.utils.prune provide built-in functions for magnitude pruning.
• Process: Typically involves training a model, pruning a target percentage of smallest weights, fine-tuning the remaining network, and repeating. This gradual approach helps recover accuracy.

A more hardware-friendly approach that removes entire structural components of a network, such as entire neurons, filters, or channels, rather than individual weights.

• Granularity: Prunes at the level of 2D convolutional filters or neuron connections, resulting in a smaller, dense model.
• Frameworks: Tools like Neural Magic's DeepSparse and NVIDIA's TensorRT have specialized support for structured sparsity patterns that align with hardware acceleration. This directly reduces the number of FLOPs and kernel calls.

Automated policies that determine the timing, rate, and final target of sparsity during the pruning process, crucial for maintaining model accuracy.

• Polynomial Decay: A common schedule that slowly increases sparsity from an initial value (e.g., 0%) to a final target (e.g., 90%) over a set number of training steps.
• Framework Support: TensorFlow's PruningSchedule API and PyTorch extensions allow engineers to define these schedules declaratively, automating the iterative prune-train cycle.

Specialized data structures used to efficiently store and compute with pruned models by only representing non-zero values and their indices.

• Compressed Sparse Row (CSR): Efficient for sparse weight matrices, storing non-zero values and column indices.
• Framework Integration: PyTorch supports sparse tensors via torch.sparse. TensorFlow uses similar representations. However, achieving speedup requires kernels optimized for these formats, which is hardware-dependent.

A training paradigm where the pruning mechanism is active during the model's training or fine-tuning phase, allowing weights to adapt to the impending sparsity.

• Lottery Ticket Hypothesis: Inspired by the finding that dense, randomly-initialized networks contain sparse, trainable subnetworks ('winning tickets'). Frameworks help identify these.
• Tool Example: The OpenLTH library provides utilities for lottery ticket hypothesis research and iterative magnitude pruning.

A compression technique that reduces the numerical precision of a neural network's weights and activations (e.g., from 32-bit floating-point to 8-bit integers). This decreases model size and memory bandwidth requirements, accelerating inference. It is often used in conjunction with pruning.

• INT8 Inference: Execution using 8-bit integers, offering major speedups on supported hardware.
• Quantization-Aware Training (QAT): Training with simulated quantization to maintain accuracy post-conversion.

A compression paradigm where a large, accurate teacher model is used to train a smaller, efficient student model. The student learns to mimic the teacher's outputs or internal feature representations, achieving comparable performance with far fewer parameters. Unlike pruning, it creates a new, dense architecture.

• Logits: The teacher's pre-softmax outputs often serve as "soft labels" for the student.
• Response-Based vs. Feature-Based: Distillation can target final predictions or intermediate layer activations.

A specialized hardware accelerator designed to execute neural network operations with extreme energy efficiency. NPUs are integral to modern smartphones and edge devices. They often include dedicated hardware support for sparse matrix operations, making them ideal for executing pruned models.

• Sparsity Support: High-end NPUs can skip computations involving zeroed weights from pruning.
• System-on-a-Chip (SoC): NPUs are typically integrated alongside CPUs and GPUs in a single chip.

An automated process for discovering optimal neural network architectures under specific hardware constraints (e.g., latency, memory). NAS can inherently design models with efficient, prune-friendly structures.

• Search Space: Defines the possible operations (e.g., depthwise convolutions) and connections.
• Controller: An RL or gradient-based algorithm that explores the space.
• Efficiency Objectives: Directly optimizes for metrics like FLOPS or measured on-device latency, often yielding sparse-friendly designs.