Performance Model

Performance models are built using two primary methodologies. Analytical models are derived from first principles, using mathematical formulas based on hardware specifications (e.g., peak FLOPS, memory bandwidth) and kernel properties (e.g., operation count, data movement). Machine-learned models (often regression-based or neural networks) are trained on empirical profiling data to capture complex, non-linear interactions between parameters and performance that are difficult to model analytically. The choice depends on the need for interpretability versus accuracy for highly irregular workloads.

A robust performance model ingests a specific set of inputs to generate its prediction. Critical inputs include:

• Kernel Parameters: Workgroup size, tile dimensions, loop unroll factors, and vectorization width.
• Hardware Characteristics: Peak compute throughput (TOPS), memory hierarchy latencies and bandwidths, cache sizes, and the number of parallel execution units.
• Workload Properties: Input tensor dimensions, batch size, sparsity pattern, and numerical precision (FP32, FP16, INT8).
• System State: Concurrent kernel execution, memory contention, and thermal throttling conditions.

The model predicts one or more key performance indicators (KPIs) essential for optimization and scheduling:

• Execution Time (Latency): The total wall-clock time to complete the kernel, often broken down into compute and memory phases.
• Compute Throughput: The achieved rate of operations (e.g., FLOPS), expressed as a percentage of the hardware's peak capability.
• Memory Bandwidth Utilization: The effective data transfer rate between memory levels, indicating if the kernel is memory-bound.
• Resource Occupancy: The utilization of parallel execution units (e.g., tensor cores, ALUs).
• Power Consumption: Estimated energy usage, crucial for edge and embedded NPU deployments.

Performance models are the engine of auto-tuning systems. They enable efficient parameter search across a vast configuration space without requiring exhaustive, time-consuming physical execution of every variant. Techniques like Bayesian optimization use the model as a surrogate to intelligently propose the most promising kernel configurations to test next, dramatically reducing the search time needed to find optimal settings for tile size selection, workgroup size, and other tunable parameters.

These models are applied throughout the NPU software stack:

• Compiler Optimization: Guiding graph compiler decisions on kernel fusion, operator scheduling, and memory allocation.
• Runtime Scheduling: Informing a runtime scheduler which kernel variant to select dynamically based on current input size or system load.
• Performance Debugging: Identifying the root cause of performance issues by comparing predicted vs. actual metrics, pinpointing bottlenecks.
• Design Space Exploration: Allowing hardware architects to evaluate the performance impact of proposed NPU features (e.g., larger cache, new instruction) before silicon fabrication.

A model's utility is determined by its fidelity—how closely its predictions match real-world measurements. Validation involves:

• Hold-out Testing: Evaluating the model on a separate dataset of profiled kernels not used during its construction.
• Error Metrics: Calculating mean absolute percentage error (MAPE) or root mean square error (RMSE) for execution time predictions.
• Continuous Calibration: Updating the model with new profiling data from field deployments to account for driver updates, hardware aging, or previously unseen workload types. High-fidelity models are essential for trustworthy auto-tuning.

These tools automate the search for optimal kernel parameters by using a performance model to predict execution time without exhaustive compilation and measurement. They define a configuration space of tunable parameters (e.g., tile size, workgroup size) and employ search algorithms like Bayesian optimization to efficiently find the best-performing variant.

• Examples: TVM's AutoTVM, OpenTuner, Kernel Tuner.
• Key Function: They replace brute-force search with guided, model-driven exploration, drastically reducing the time required for performance optimization.

Modern machine learning compilers embed analytical performance models directly into their cost models to make optimal graph-level and kernel-level optimization decisions. The compiler uses the model to evaluate the cost of different graph compilation strategies, such as operator fusion or layout transformations, before generating code.

• Examples: Apache TVM, MLIR-based compilers, XLA.
• Key Function: Enables hardware-aware model optimization by predicting the latency and memory usage of different compiled versions of a neural network graph for a target NPU.

Hardware vendors provide tools that combine empirical profiling data with built-in performance models to offer optimization recommendations. These tools analyze execution traces and performance counter data (e.g., cache hit rate, memory bandwidth) to identify bottlenecks and suggest specific code changes.

• Examples: NVIDIA Nsight Compute, AMD ROCProfiler, Intel VTune Profiler.
• Key Function: They bridge the gap between raw hardware events and actionable insights, often using models to explain why a kernel is compute bound or memory bound.

In cluster and cloud environments, performance models are used by schedulers to predict job completion times and optimize resource allocation. By estimating the runtime of a kernel or containerized workload based on its characteristics and available hardware, schedulers can improve throughput and reduce resource contention.

• Use Case: Kubernetes schedulers with custom metrics, HPC job schedulers (SLURM) with performance plugins.
• Key Function: Enables efficient heterogeneous fleet orchestration by matching workloads to the most suitable NPU or GPU instance type.

Emerging cloud services offer performance prediction for machine learning workloads as an API. Users submit kernel code or a model graph, and the service returns predicted latency, throughput, and cost across a range of hardware targets, using a backend database of pre-characterized performance models.

• Concept: Similar to cost estimation for cloud compute, but for inference latency and FLOPs efficiency.
• Key Function: Allows architects to perform bottleneck analysis and capacity planning before any code is deployed or hardware is provisioned.

Performance models are incorporated into continuous integration systems to guard against performance regressions. A lightweight model can predict the runtime of key kernels from code changes. If the prediction exceeds a threshold, the build can be flagged for deeper profiling with a kernel profiler or sampling profiler.

• Practice: Part of MLOps and Evaluation-Driven Development.
• Key Function: Provides fast, automated performance regression testing, ensuring optimization efforts are not inadvertently undone by new commits.

A software tool that measures the execution time, resource utilization, and hardware performance counters of a computational kernel running on an NPU. It provides the raw empirical data used to validate and refine analytical performance models.

• Key Outputs: Kernel duration, compute throughput (FLOPS), memory bandwidth utilization, cache hit/miss rates.
• Types: Sampling profilers (low-overhead, statistical) and instrumentation profilers (high-precision, intrusive).
• Use Case: Identifying if a kernel is compute bound or memory bound, and pinpointing hotspots for optimization.

The automated process of searching a configuration space of kernel parameters to find the optimal setup for a specific NPU. It uses empirical feedback (often from a profiler) to navigate the search space, effectively operationalizing a performance model.

• Search Methods: Include exhaustive search, random search, and model-guided approaches like Bayesian optimization.
• Tunable Parameters: Workgroup size, tile size, loop unrolling factor, and vectorization factor.
• Goal: To overcome the complexity of manually modeling every hardware interaction by letting the system discover the best configuration through guided experimentation.

The systematic process of identifying the primary limiting factor that restricts the performance of an NPU kernel. A performance model's predictions are used to hypothesize bottlenecks, which are then verified with profiling data.

• Common Bottlenecks:
  • Memory Bound: Kernel stalls waiting for data from DRAM or cache.
  • Compute Bound: ALUs are saturated; performance limited by arithmetic throughput.
  • Latency Bound: Execution suffers from pipeline stalls or high-latency operations.
• Tools: Profilers and execution traces are used to measure metrics like occupancy and memory coalescing efficiency to pinpoint the root cause.

A chronological, detailed record of the low-level operations performed during kernel execution on an NPU. Unlike summary profiling, a trace provides a timeline view of instructions, memory accesses, and synchronization events.

• Provides Insight Into:
  • The exact sequence of operations leading to a pipeline stall.
  • Patterns of thread divergence within warps.
  • Asynchronous execution and concurrent kernel overlap.
• Use Case: Debugging complex performance issues where aggregate metrics are insufficient, such as analyzing resource contention or irregular memory access patterns.

The two fundamental hardware ceilings that a performance model must account for. Compute throughput (e.g., Tera-Operations Per Second - TOPS) and memory bandwidth (GB/s) define the theoretical peak performance of an NPU.

• Roofline Model: A canonical performance model that plots attainable performance (FLOPS) against operational intensity (Ops/Byte), bounded by these two ceilings.
• Arithmetic Intensity: A kernel's ratio of operations to bytes moved from memory; determines which ceiling (compute or bandwidth) it will hit first.
• Engineering Goal: Transform kernels (via tiling, fusion) to increase arithmetic intensity and push performance toward the compute roof.

The multidimensional set of all possible combinations of tunable parameters for a kernel. Defining and navigating this space is central to auto-tuning and is implicitly modeled by learned performance models.

• Parameters Include:
  • Thread block/grid dimensions (workgroup size).
  • Loop tile sizes for cache locality.
  • Degrees of loop unrolling and vectorization.
  • Memory padding and prefetch settings.
• Challenge: The space is vast and non-linear; performance can change abruptly with small parameter changes.
• Solution: Use parameter search strategies and surrogate performance models to efficiently explore high-performing regions.