Compute Throughput

Floating-Point Operations Per Second (FLOPS) and Trillions of Operations Per Second (TOPS) are the standard units for measuring compute throughput. FLOPS is precise, counting the number of 32-bit (FP32) or 16-bit (FP16) floating-point calculations completed each second. TOPS is a more generalized unit, often used for integer operations (e.g., INT8) common in inference. A key distinction is between peak theoretical throughput (hardware limit under ideal conditions) and achieved throughput (actual performance on a real workload, often lower due to memory bottlenecks).

Throughput is often limited not by compute but by the speed of data movement. Memory Bandwidth, measured in GB/s (Gigabytes per second), is the maximum rate data can be transferred between the NPU's compute cores and its memory hierarchy (e.g., HBM, SRAM). A workload is memory-bound if the compute units are idle waiting for data. The Arithmetic Intensity (FLOPs per byte) of a kernel determines if it is compute-bound or memory-bound. High-bandwidth memory (HBM) is critical for achieving high sustained TOPS.

These metrics measure how effectively the hardware's theoretical throughput is being used.

• SM/TPU Utilization: Percentage of time the NPU's streaming multiprocessors or tensor cores are actively executing instructions.
• Occupancy: The ratio of active warps/wavefronts to the maximum supported on a compute unit. High occupancy helps hide memory latency.
• Cache Hit Rate: The percentage of memory requests served from fast cache (L1, L2) versus slow global memory. High rates reduce effective memory latency and improve throughput.

Throughput (operations/second) and Latency (seconds/operation) are inversely related but distinct. High throughput processes many operations in parallel (e.g., batch inference), while low latency minimizes the time for a single operation (e.g., real-time response). Optimizations differ:

• Throughput: Maximize batch size, use kernel fusion, enable concurrent kernel execution.
• Latency: Minimize data dependencies, optimize critical path, use smaller efficient models. System design must balance these based on the application's Service Level Objective (SLO).

Vendor peak TOPS are rarely achieved. Industry benchmarks measure sustained throughput on representative workloads:

• MLPerf Inference: Standardized suite measuring latency and throughput for models like ResNet-50 and BERT across different scenarios (Offline, Server, Single-Stream, Multi-Stream).
• Throughput is reported as samples/second or queries/second.
• Performance/Watt (samples/Joule) is a critical derived metric for edge deployment, combining throughput with power efficiency.

Identifying the limiting factor is essential for optimization. Profiling tools measure:

• Compute-Bound: High ALU utilization but low memory traffic. Solution: Increase arithmetic intensity, use lower precision (FP16/INT8).
• Memory-Bound: High memory traffic, low ALU utilization. Solution: Improve data locality (tiling), increase cache hit rate, fuse kernels to reduce intermediate data writes.
• Latency-Bound: Stalls due to pipeline dependencies or branch divergence. Solution: Restructure algorithms, increase occupancy to hide latency. Tools like kernel profilers and performance counters provide the data for this analysis.

The maximum rate at which data can be read from or written to a memory subsystem (e.g., High Bandwidth Memory, SRAM) by the NPU. It is measured in gigabytes per second (GB/s).

• Critical Bottleneck: Most neural network layers are memory-bound, meaning the compute units idle while waiting for weights and activations.
• Bandwidth vs. Throughput: Peak theoretical FLOPS (compute throughput) is only achievable if memory bandwidth can supply data fast enough. The roofline model visualizes this relationship.
• Optimization: Techniques like memory coalescing, prefetching, and kernel fusion aim to reduce off-chip memory accesses and maximize effective bandwidth utilization.

The two fundamental operational states that determine the limiting factor for a kernel's execution time.

• Compute-Bound: Execution time is limited by the NPU's arithmetic logic unit (ALU) speed. The kernel performs dense mathematical operations (e.g., large matrix multiplies) with high arithmetic intensity. Optimizations focus on maximizing ALU utilization and occupancy.
• Memory-Bound: Execution time is limited by memory subsystem speed. The kernel has low arithmetic intensity and spends most cycles waiting for data. Optimizations focus on improving cache hit rates, data reuse, and reducing redundant transfers.
• Analysis: Bottleneck analysis using performance counters identifies which state applies, directing optimization efforts.

The ratio of actively executing warps (or wavefronts) to the maximum number that can be resident on an NPU's streaming multiprocessor (SM). It measures the utilization of parallel execution resources.

• Resource Limits: Occupancy is limited by register file size, shared memory allocation, and workgroup size.
• Performance Impact: High occupancy helps hide memory latency by allowing the scheduler to switch to other ready warps while one waits for data. However, beyond a point, instruction-level parallelism and memory bandwidth become the limiting factors.
• Tuning: Auto-tuning tools often sweep workgroup size and resource usage to find the optimal balance between occupancy and other performance factors.

A chronological, detailed record of the micro-ops, instructions, memory accesses, and control flow executed by a kernel on the NPU hardware. It provides a ground-truth view of runtime behavior.

• Granular Insight: Goes beyond aggregate metrics to show pipeline stalls, memory access patterns, thread divergence, and exact instruction scheduling.
• Use Case: Essential for hotspot identification at the instruction level and for validating the assumptions of performance models. It explains why a particular code section is slow.
• Tooling: Generated by low-level hardware profilers or simulators, often visualized in timeline tools to correlate kernel activity with performance counter data.

The automated process of searching a configuration space of kernel parameters to find the optimal setup for a specific NPU and workload. It abstracts hardware complexity from the developer.

• Tunable Parameters: Includes workgroup size, tile size, loop unrolling factor, vectorization factor, and shared memory allocation.
• Search Strategies: Methods range from exhaustive search (for small spaces) to guided searches like Bayesian optimization or genetic algorithms. A performance model can prune the search space.
• Output: The kernel tuner produces a set of optimal parameters or even a generated, optimized kernel variant, directly maximizing compute throughput and minimizing latency.

An analytical or machine-learned representation that predicts the execution time, resource usage, or throughput of a computational kernel based on its parameters, input data, and target hardware characteristics.

• Purpose: Guides auto-tuning by predicting performance without requiring exhaustive, time-consuming execution of every configuration. Used for bottleneck analysis and compiler optimization decisions.
• Types: Can be simple (e.g., based on roofline model arithmetic intensity) or complex (e.g., a neural network trained on execution trace and performance counter data).
• Application: In graph compilers, performance models decide operator fusion strategies and scheduling to maximize overall pipeline throughput.