Weight Pruning

A compression technique that reduces the numerical precision of weights and activations from 32-bit floating point to 8-bit integers or lower. Unlike pruning, which removes parameters entirely, quantization shrinks each remaining parameter's bit width.

• Post-Training Quantization (PTQ): Applied after training with minimal calibration data
• Quantization-Aware Training (QAT): Simulates quantization during training to preserve accuracy
• Typical formats: INT8, FP16, INT4

Combined with pruning, quantization can yield 10-20x total model size reduction with negligible accuracy loss.

A training paradigm where a compact student model learns to mimic the output distribution of a larger, more accurate teacher model. The student is trained on soft labels—the teacher's probability outputs—rather than hard ground-truth labels.

• Captures inter-class relationships that one-hot labels miss
• Often combined with pruning to recover accuracy lost during sparsification
• Enables 5-50x compression while retaining teacher-level performance

Distillation is particularly effective when pruning removes entire channels or layers, as the student can learn to compensate for lost representational capacity.

A methodology that incorporates target hardware constraints—latency, energy, memory bandwidth—directly into the model optimization loop. Rather than optimizing for abstract metrics like FLOPs, hardware-aware training uses profiled cost models.

• Neural Architecture Search (NAS) with latency constraints
• Differentiable pruning masks that account for hardware sparsity patterns
• Platform-specific operator fusion and tiling strategies

This ensures pruned models achieve actual speedup on real silicon, not just theoretical parameter reduction. Unstructured sparsity often fails to accelerate inference without hardware-aware design.

A graph optimization that merges multiple discrete operations into a single compute kernel to eliminate intermediate memory reads and writes. Common fusions include:

• Conv + BatchNorm + ReLU into a single fused kernel
• MatMul + Bias + Activation for transformer feed-forward layers
• Element-wise operation chains (Add, Mul, Sigmoid)

Fusion complements pruning by reducing the memory bandwidth bottleneck that often dominates inference time. When pruning removes entire operations, fusion opportunities may change, requiring re-optimization of the compute graph.

NVIDIA's inference compiler and runtime that applies a suite of optimizations including layer fusion, precision calibration, kernel auto-tuning, and dynamic tensor memory management.

• Automatically exploits structured sparsity (2:4 pattern) on Ampere+ GPUs
• Converts pruned models to efficient sparse matrix formats
• Delivers 2-3x throughput improvement on sparse models vs. dense equivalents

TensorRT bridges the gap between theoretical sparsity from pruning and measurable inference acceleration by mapping sparse weights to hardware-accelerated sparse tensor cores.

The empirical finding that dense, randomly-initialized networks contain sparse subnetworks—winning tickets—that can be trained in isolation to match the full network's accuracy. Key implications for pruning:

• Iterative Magnitude Pruning (IMP) identifies these subnetworks
• Winning tickets exhibit commensurate learning rates with the original network
• Suggests overparameterization aids optimization, not just representational capacity

This hypothesis reframes pruning as a training-aware search problem rather than a post-hoc compression step, influencing modern sparse training techniques.