Sparse Model Profiling

The percentage of zero-valued parameters within each layer of a neural network. This is the foundational metric for sparse profiling.

• Key Insight: Not all sparsity is equal. A model with 90% overall sparsity may have critical layers with only 50% sparsity, creating a performance bottleneck.
• Measurement: Calculated as (1 - (nnz(weights) / total_weights)) * 100% for each layer.
• Profiling Use: Guides optimization efforts by identifying layers where further pruning may yield the greatest computational savings or where sparsity is too low to benefit from specialized kernels.

A comparison between the actual floating-point operations (FLOPs) executed with zero-skipping and the theoretical FLOPs required if the model were dense.

• Sparse FLOPs: Counts only multiplications and additions involving non-zero weights and activations. This represents the theoretical lower bound of compute work.
• Theoretical Speedup: Calculated as Dense FLOPs / Sparse FLOPs. A 10x reduction in FLOPs suggests a potential 10x speedup.
• Reality Check: This metric alone is optimistic; actual speedup is governed by memory bandwidth and kernel overhead, leading to the sparse efficiency gap.

Measures the effective data throughput between the processor and memory during sparse inference, often the true limiting factor.

• The Bottleneck: Sparse computations are typically memory-bound, not compute-bound. The irregular access patterns of gather-scatter operations can lead to poor cache efficiency and high DRAM traffic.
• Profiling Data: Tools like NVIDIA Nsight Systems or rocm-smi track memory read/write volumes. High bandwidth usage relative to the hardware's peak indicates efficient data feeding; low usage indicates stalls due to irregular access.
• Optimization Target: Improving data locality through sparse data layout optimizations (e.g., blocked formats) aims to increase this metric.

The frequency at which requested data is not found in the processor's fast cache memory, necessitating a slower fetch from main memory.

• Critical for Performance: High cache miss rates are catastrophic for sparse kernels due to the random, indirect memory accesses required to fetch non-zero weights and corresponding activations.
• Causes: Irregular sparsity patterns (unstructured pruning) and suboptimal sparse data layouts (e.g., naive COO) exacerbate this problem.
• Profiling Method: Hardware performance counters (e.g., perf on Linux) provide L1/L2/L3 cache miss statistics. Reducing this rate is a primary goal of format conversion and kernel optimization.

The wall-clock time to execute a sparse kernel and the variance in work assigned across parallel threads.

• Execution Time: The ultimate bottom-line metric for a sparse operator (e.g., SpMM kernel). Must be compared against its dense counterpart to measure real speedup.
• Load Imbalance: A major performance killer in parallel sparse computation. If non-zero elements are clustered, some GPU warps or CPU cores finish quickly while others are overloaded, leaving hardware idle.
• Profiling Insight: Tools like NVIDIA Nsight Compute reveal thread divergence and stall reasons. Structured sparsity patterns like N:M sparsity are designed specifically to mitigate this imbalance.

The reduction in model storage size achieved by sparsity, calculated relative to the dense model's size.

• Calculation: Size(Dense Model) / Size(Sparse Model). Size includes both the quantized weights and the necessary indexing metadata (e.g., CSR format indices, bitmask encoding).
• Overhead: The pruning mask or index storage can offset gains. A model with 90% weight sparsity may only achieve a 5x compression ratio due to this metadata.
• System Impact: Directly affects DRAM footprint, I/O time for loading the model, and last-level cache pressure. It is a key metric for on-device model formats.

A family of data structures for efficiently storing and operating on tensors where most elements are zero. Instead of storing all values, they encode only the non-zero values and their indices. Common formats include:

• CSR (Compressed Sparse Row): Optimized for row-wise operations; stores column indices, values, and compressed row pointers.
• CSC (Compressed Sparse Column): Column-oriented analog of CSR.
• COO (Coordinate Format): Simple list of (row, column, value) tuples for each non-zero. The choice of format is a primary factor in profiling, as it directly impacts memory bandwidth, cache locality, and the efficiency of gather-scatter operations during kernel execution.

The fundamental computational kernel for executing sparse linear layers, performing the operation Y = Sparse_Matrix * Dense_Matrix. Profiling focuses on the efficiency of its zero-skipping implementation. Key performance determinants include:

• Sparse data layout (e.g., CSR, Blocked) and its alignment with hardware memory access patterns.
• Overhead from index decoding and pointer chasing.
• Degree of load imbalance across parallel threads due to irregular non-zero distribution. Optimized SpMM kernels in frameworks like PyTorch or vendor SDKs are critical for realizing the theoretical FLOP reduction from sparsity.

The observed performance delta between the theoretical speedup predicted by the reduction in FLOPs and the actual speedup achieved on hardware. A core focus of sparse model profiling is to diagnose the causes of this gap, which include:

• Sparse kernel overhead: Costs from conditional branches, metadata processing, and gather-scatter instructions.
• Memory bandwidth saturation: Irregular access patterns can lead to poor cache utilization and become bandwidth-bound.
• Unstructured sparsity: Fine-grained, irregular pruning patterns often yield high theoretical sparsity but poor realized speedup on general-purpose hardware without native sparse support.

Two pruning methodologies that create sparsity patterns with vastly different profiling characteristics.

• Structured Pruning: Removes entire structural components (e.g., filters, channels, layers). Results in regular sparsity that is easily mapped to dense hardware operations, leading to predictable speedups but potentially greater accuracy loss.
• Unstructured Pruning: Removes individual weights based on saliency (e.g., magnitude-based pruning). Creates irregular, fine-grained sparsity that maximizes parameter reduction but requires specialized kernels (SpMM) and hardware (e.g., Sparse Tensor Cores) for efficient execution. Profiling must account for the associated overhead.

Specialized hardware units in modern NVIDIA GPUs (Ampere architecture and later) designed to accelerate sparse matrix operations. They leverage a specific 2:4 sparsity pattern (2 non-zeroes in every block of 4 values). Profiling for this hardware involves:

• Ensuring the model's sparsity matches the 2:4 structured pattern.
• Measuring the effective doubling of theoretical dense TOPS (Tera Operations Per Second).
• Analyzing compiler transformations that map sparse operations to these units via sparse hardware mapping. This represents a case where profiling guides model compression to align with hardware capabilities.

Fundamental parallel computing primitives critical for sparse computation performance.

• Gather: Reads a set of non-contiguous data elements (e.g., non-zero activations) into a contiguous vector for computation.
• Scatter: Writes a contiguous result vector back to non-contiguous memory locations. In profiling, these operations are often the bottleneck. Their cost is determined by:
• The degree of memory access irregularity.
• Cache miss rates caused by non-localized reads.
• Hardware support for gather-scatter instructions. Efficient kernels minimize the latency of these data-movement operations.