Weight Pruning

The most common heuristic where weights with the smallest absolute values are removed, based on the principle that near-zero weights contribute negligibly to network output. Unstructured pruning zeroes out individual weights regardless of position, while structured pruning removes entire neurons, channels, or filters to maintain dense matrix operations. Post-pruning, the model typically requires fine-tuning to recover accuracy lost from the removed connections.

Unstructured sparsity removes individual weights, creating irregular zero patterns that require specialized sparse matrix libraries or hardware to accelerate. Structured sparsity removes contiguous blocks—entire channels, filters, or attention heads—producing models that run efficiently on standard dense hardware without custom kernels. The trade-off: unstructured achieves higher compression with minimal accuracy loss, while structured delivers immediate speedup on commodity accelerators.

Unlike magnitude pruning which relies on weight values at a single point, movement pruning scores weights based on how they change during fine-tuning. Weights that move consistently away from zero are retained; those that oscillate or remain near zero are pruned. This method is particularly effective for compressing large pre-trained models adapted to specific downstream tasks, often outperforming magnitude-based approaches for transformer architectures.

The pruning schedule defines when and how aggressively weights are removed during training. Gradual magnitude pruning starts from a dense network and progressively increases sparsity throughout training, allowing the remaining weights to adapt. Common schedules include cubic, exponential, and polynomial decay functions. A well-tuned schedule prevents catastrophic accuracy collapse by giving the network time to redistribute representational capacity before the next pruning step.

A training paradigm where a compact student model learns to replicate the behavior of a larger, more accurate teacher model. The student is trained on the teacher's soft output probabilities rather than hard labels, transferring dark knowledge about inter-class relationships.

• Produces dense, efficient models rather than sparse pruned ones
• Often combined with pruning for maximum compression
• Student architectures can be fundamentally different from teacher
• Preserves decision boundaries that pruning alone might distort

A compiler optimization that merges multiple discrete neural network operations into a single kernel launch. For pruned models with irregular sparsity patterns, custom fused kernels are essential to translate theoretical parameter reduction into actual speedup.

• Eliminates intermediate memory reads and writes
• Reduces kernel launch overhead on GPU and NPU
• Critical for realizing latency gains from unstructured pruning
• Requires sparsity-aware compiler backends

The technique of splitting a neural network's computational graph across multiple processing units or edge nodes. Pruning can create uneven compute distributions, requiring intelligent partitioning strategies to balance load.

• Horizontal partitioning splits layers across devices
• Vertical partitioning pipelines sequential layers
• Pruned subgraphs may fit entirely on smaller accelerators
• Enables deployment of models too large for any single edge device

A guaranteed maximum time window within which inference must complete, essential for closed-loop industrial control. Pruning reduces worst-case compute paths, but unstructured sparsity can introduce variance that challenges determinism guarantees.

• Structured pruning preferred for hard real-time systems
• Unstructured pruning requires careful kernel engineering
• RTOS scheduling must account for sparse computation patterns
• Safety Integrity Level (SIL) certification demands bounded execution