Structured Pruning

Structured pruning operates at the channel or filter level, removing entire feature map dimensions from convolutional layers. Unlike unstructured pruning, which zeroes out individual weights, this technique eliminates whole kernels and their associated input/output connections. The result is a physically smaller weight tensor that reduces both memory footprint and multiply-accumulate operations (MACs) in direct proportion to the channels removed.

Because structured pruning produces a dense sub-network rather than a sparse one, the compressed model runs efficiently on commodity hardware without custom sparse matrix libraries. Standard SIMD instructions, GPU tensor cores, and NPU accelerators all benefit from the reduced channel count. This is the primary advantage over unstructured pruning: the speedup is realized on off-the-shelf inference engines like TensorRT, OpenVINO, and Core ML without specialized software support.

Channels are selected for removal based on an importance score. Common criteria include:

• L1-norm of filter weights: smaller magnitude filters are pruned first
• Batch Normalization scaling factor (γ): channels with small γ values are treated as less critical
• Taylor expansion: estimates the impact of removing a channel on the loss function
• Activation-based ranking: channels with consistently low average activations across the calibration dataset are pruned

Aggressive one-shot pruning often causes catastrophic accuracy collapse. Best practice employs an iterative schedule: prune a small percentage of channels, then fine-tune the remaining weights for several epochs to recover accuracy. This prune-then-retrain cycle repeats until the target compression ratio is met. The learning rate during fine-tuning is typically set to 1/10th of the original training rate to avoid destabilizing the converged weights.

Modern structured pruning frameworks like Torch-Pruning and NNI build a dependency graph of the model's computational graph before pruning. This ensures that when a channel is removed from one layer, the corresponding input channels in downstream layers and output channels in upstream layers are pruned simultaneously. Without dependency resolution, dimension mismatches would break the forward pass. This is especially critical in architectures with skip connections and concatenation operations.

Structured pruning typically achieves 2× to 5× compression with less than 1% accuracy degradation on classification tasks. For dense prediction tasks like medical image segmentation, the tolerance is lower—pruning beyond 30% of channels often degrades Dice score on small anatomical structures. The trade-off curve is architecture-dependent: ResNet and VGG prune gracefully, while compact architectures like MobileNet have less redundancy and are more sensitive to channel removal.

The counterpart to structured pruning, this method zeroes out individual weights based on a magnitude or saliency criterion, creating a sparse weight matrix. Unlike structured pruning, it does not remove entire channels or layers, meaning the resulting model has an irregular sparsity pattern. This requires specialized sparse linear algebra libraries or custom hardware to realize actual speedups. In diagnostic imaging, unstructured pruning can achieve higher compression ratios with less accuracy loss, but the inference speedup on standard edge hardware like a Jetson Orin is often negligible without sparsity-aware runtimes.

A compression technique that reduces the numerical precision of weights and activations from 32-bit floating point (FP32) to lower bit-widths like INT8 or FP16. Quantization is orthogonal and complementary to structured pruning; a pruned model can be further quantized for compounded benefits. Post-Training Quantization (PTQ) requires only a small calibration dataset, while Quantization-Aware Training (QAT) simulates quantization during training for higher final accuracy. For edge-deployed diagnostic models, INT8 quantization on a Neural Processing Unit (NPU) can yield 2-4x latency improvements with minimal accuracy degradation.

A training paradigm where a compact student model is trained to mimic the output distribution of a larger, high-capacity teacher model. The student learns not just from hard labels but from the teacher's softened logits, capturing inter-class relationships. This is often used as a precursor to structured pruning: a distilled student provides a better initialization for pruning, or the distillation loss can guide which structures to remove. In medical imaging, a ResNet-50 student distilled from an ensemble of vision transformers can match diagnostic accuracy while being amenable to structured pruning for edge deployment.

A model optimization paradigm that incorporates target hardware constraints directly into the training or pruning objective. Instead of optimizing solely for accuracy, the loss function includes differentiable proxies for latency, energy consumption, or memory footprint on the specific edge accelerator. This enables co-optimization of the model architecture and the pruning mask for a given device like an FPGA or ASIC. For scanner-side AI, hardware-aware structured pruning can produce models that meet strict real-time constraints, such as sub-50ms inference for CT reconstruction on a specific medical-grade system-on-module.

A specialized runtime environment that loads a compressed model and executes its computational graph on target hardware. Engines like ONNX Runtime, OpenVINO, and TensorRT each have unique strengths for structured-pruned models. ONNX Runtime provides broad hardware support via execution providers; OpenVINO optimizes for Intel CPUs and VPUs; TensorRT delivers peak performance on NVIDIA GPUs. The choice of inference engine directly impacts whether the theoretical speedup from structured pruning translates into wall-clock latency reduction on the deployed medical device.