Activation Sparsity

Activation sparsity is dynamic, meaning the pattern of zero values changes with every input sample during inference. This contrasts with weight sparsity, which is static and fixed after pruning. The dynamic nature makes activation sparsity harder to predict and optimize for, as the computational savings are data-dependent. For example, a ReLU layer will produce different zero patterns for an image of a cat versus a dog.

Sparsity in activations is primarily induced by non-linear activation functions that map some inputs to zero. The most common source is the Rectified Linear Unit (ReLU), which outputs max(0, x). Other functions like Leaky ReLU and Sigmoid do not create exact zeros. The choice of activation function directly controls the sparsity ratio and the potential for computational skipping in subsequent layers.

Activation sparsity is not isolated; it propagates through the network. The zeros in the output of one layer become ineffective operands for the next layer's computations.

• In a fully connected layer, a zero activation eliminates an entire vector dot product.
• In a convolutional layer, a zero in an input feature map eliminates all multiplications for that spatial position across all output channels. This cascading effect can lead to significant, compound reductions in actual floating-point operations (Sparse FLOPs).

Realizing speedups from activation sparsity requires sparse inference kernels (e.g., Sparse Matrix Multiplication (SpMM) or Sparse Convolution). These kernels must:

• Skip computations where an activation or weight is zero.
• Efficiently handle irregular memory access patterns via gather-scatter operations.
• Manage load imbalance, where threads have uneven work due to unpredictable zero distributions. Naive execution on dense hardware forfeits the potential acceleration, as zeros are still fetched and multiplied.

The benefit is governed by two metrics:

• Sparsity Ratio: The percentage of activations that are zero (e.g., 70%). Higher ratios enable greater skipping.
• Spatial Distribution: How zeros are clustered. Structured blocks of zeros (e.g., entire channels) are more efficient to exploit than unstructured, random zeros, as they lead to more regular memory access and less kernel overhead. The effective speedup is often less than the sparsity ratio due to this overhead and memory bandwidth limits.

Activation and weight sparsity can be composed for multiplicative efficiency gains. When a sparse weight meets a sparse activation, the operation is skipped entirely. This is the target of sparse-sparse kernels. However, exploiting combined sparsity is algorithmically and architecturally more complex. Modern techniques like N:M sparsity patterns in weights are designed to be efficiently executed on hardware (e.g., Sparse Tensor Cores) and can compound benefits with dynamic activation sparsity.

Scalable Vector Extensions v2 (SVE2) and Scalable Matrix Extensions (SME) for ARMv9-A processors introduce instructions beneficial for sparse workloads. Key features include:

• Gather-load and scatter-store instructions for efficient non-contiguous data access.
• Vector predicate registers to mask operations on zero elements.
• SME's outer product instructions can be optimized to skip zero-rich slices of input matrices. These capabilities are foundational for efficient sparse inference on future mobile SoCs and server CPUs.

Intel's Advanced Matrix Extensions (AMX) and AVX-512 instruction sets provide the basis for sparse acceleration on X86 CPUs.

• AMX features tile registers and instructions (TDPBF16PS) that can be paired with sparse data layouts to skip computations on zero tiles.
• AVX-512 includes mask registers and compressed load/store instructions (VPCOMPRESS) that are essential for implementing efficient zero-skipping kernels. The VPOPCNTD instruction rapidly counts non-zero bits in sparsity bitmasks.

Many dedicated Neural Processing Units (NPUs) and AI accelerators incorporate first-class support for sparsity in their microarchitecture.

• Google's TPU v4/v5e uses sparse core subsystems designed for ultra-low-precision and sparse computations.
• Apple Neural Engine likely employs hardware-based zero-gating to skip MAC operations where activations are zero.
• Startups like Groq and Tenstorrent design their deterministic execution models and memory hierarchies to minimize the overhead of load imbalance inherent in sparse workloads.

The property of a neural network where a significant fraction of its weight parameters are exactly zero, typically induced through pruning algorithms. Unlike activation sparsity, which is dynamic per input, weight sparsity is static after training.

• Key Difference: Static (weights) vs. Dynamic (activations).
• Primary Benefit: Reduces model size and the number of multiplications required during inference.
• Induction Method: Achieved via pruning techniques like magnitude-based pruning or regularization during training.

A family of data structures for efficiently storing and operating on tensors where most elements are zero. These formats store only non-zero values and their indices, critical for exploiting both weight and activation sparsity.

• Common Formats: CSR (Compressed Sparse Row) for 2D matrices; COO (Coordinate Format) for simplicity; specialized blocked formats for GPUs.
• Memory Trade-off: Saves space on values but adds overhead for index data.
• Runtime Impact: The chosen format dictates the efficiency of sparse kernels due to memory access patterns.

The fundamental computational kernel for multiplying a sparse matrix by a dense matrix. It is the core operation in executing a pruned linear layer or convolution. Efficiency requires skipping multiplications with zero elements.

• Algorithm Challenge: Managing irregular, data-dependent memory access patterns.
• Key Primitives: Relies heavily on gather-scatter operations to collect non-contiguous data.
• Performance Limiter: Often bound by memory bandwidth, not compute, due to the overhead of processing indices.

Specialized hardware units in modern NVIDIA GPUs (Ampere architecture and later) designed to accelerate sparse matrix operations. They exploit a specific structured sparsity pattern (2:4) to effectively double theoretical compute throughput.

• Pattern Requirement: Requires weights to have the N:M sparsity pattern (e.g., 2 non-zeros in every block of 4).
• How it Works: The hardware skips computations on zero values at the warp level, maintaining dense execution pipelines.
• Use Case: Primarily accelerates inference for models pruned to meet the 2:4 structural constraint.

The foundational runtime optimization in sparse inference where computations involving zero-valued operands (weights or activations) are identified and omitted. The actual speedup depends on the efficiency of the skipping logic.

• Theoretical vs. Real Speedup: Reduces FLOPs, but overhead can cause a sparse efficiency gap.
• Implementation: Requires conditional checks or bitmask encoding to identify zeros, which introduces branch instructions.
• Goal: To translate reduced FLOPs into reduced latency and energy consumption.

Critical parallel computing primitives for sparse computation. Gather loads non-contiguous data (e.g., non-zero activations) into a contiguous register, and Scatter writes results back to non-contiguous locations.

• Role in SpMM: Gather fetches the specific dense matrix rows/columns needed for multiplication with non-zero weights.
• Performance Impact: These operations are often memory-intensive and can cause load imbalance across threads.
• Hardware Support: Modern architectures include instructions to accelerate these patterns.