Hotspot Identification

Hotspot identification operates across multiple levels of the execution stack, from high-level application functions down to individual hardware instructions. Instrumentation profilers and sampling profilers collect data at these different granularities. The process typically follows a top-down approach:

• Application Level: Identify slow-running modules or functions.
• Kernel Level: Isolate specific computational kernels dispatched to the NPU.
• Instruction Level: Analyze the assembly or intermediate representation for stalls, low compute throughput, or poor memory coalescing. This hierarchical view is essential for determining whether a bottleneck is algorithmic, related to kernel implementation, or a hardware limitation.

Identification relies on precise, quantitative metrics rather than intuition. Profilers measure:

• Absolute Time: Total execution time or latency of functions/kernels.
• Relative Percentage: The proportion of total runtime consumed by a hotspot (e.g., 'Function X uses 75% of total time').
• Hardware Counters: Low-level performance counter data from the NPU, such as:
  • FLOPS/TOPS: Actual achieved compute throughput.
  • Memory Bandwidth Utilization: Percentage of peak bandwidth used.
  • Cache Hit Rate: Efficiency of the memory hierarchy.
  • Occupancy: Utilization of parallel execution units. These metrics definitively prove a code section is a bottleneck and provide a baseline for measuring optimization success.

A true hotspot is characterized not just by high time consumption, but by the underlying resource constraint causing the delay. The primary classifications are:

• Compute-Bound: The hotspot is limited by the NPU's arithmetic logic units (ALUs). Symptoms include high ALU utilization but low memory traffic. Optimization focuses on increasing parallelism and instruction-level parallelism.
• Memory-Bound: The hotspot is limited by memory bandwidth or latency. Symptoms include high numbers of cache misses and memory transactions while ALUs are idle. Optimization focuses on memory hierarchy management, improving access patterns, and cache hit rate.
• Latency-Bound: The hotspot is limited by dependencies and pipeline stalls, often due to control flow issues like high thread divergence or serialized operations. Correct classification directs the optimizer to the most effective transformation strategy.

Raw profiling data is transformed into visualizations to make hotspots immediately apparent and to understand call relationships. The most critical tool is the Flame Graph, which provides a hierarchical, time-ordered view:

• The x-axis shows the stack profile population, sorted alphabetically.
• The y-axis shows stack depth.
• The width of each horizontal bar represents the time proportion spent in that function and its descendants. Wide bars at the top-level indicate major bottlenecks. Drilling down into a wide bar reveals the specific call path causing the issue. Other visualizations include timeline views for asynchronous execution and concurrent kernel analysis, and heatmaps for memory access patterns.

A hotspot is not an absolute property of code; it is contextual to the specific hardware, input data, and system state.

• Hardware Context: A loop may be memory-bound on one NPU architecture but compute-bound on another with a wider memory bus.
• Data Context: Performance characteristics can change with different input sizes or shapes, affecting tile size selection efficacy.
• System Context: Resource contention from other processes or kernels can create transient hotspots. Therefore, hotspot identification is an iterative process within the auto-tuning cycle. After an optimization is applied, profiling must be repeated to validate improvement and reveal the next limiting bottleneck, continuing the cycle of measurement and refinement.

Hotspot identification directly feeds the auto-tuning process by defining the optimization target and constraints.

• Target Kernel: Identifies which specific kernel to apply parameter search and transformations to.
• Optimization Goal: Informs the performance model—should it aim to reduce memory traffic, increase compute density, or hide latency?
• Parameter Constraints: Informs the configuration space. For example, if a kernel is memory-bound, the tuner might prioritize exploring larger tile sizes to improve locality, rather than higher vectorization factors. Without accurate hotspot identification, auto-tuning wastes cycles optimizing code sections that have negligible impact on overall performance, a classic violation of Amdahl's Law.

A classic hotspot where the kernel's execution time is limited by the speed of loading data from memory, not by the NPU's compute units. This occurs when the algorithm's arithmetic intensity (FLOPs per byte) is low.

• Symptoms: High cache miss rates, low compute unit utilization, performance scaling poorly with increased core count.
• Identification: Profilers show high memory bandwidth saturation and low compute throughput.
• Example: A naive implementation of a large matrix multiplication (GEMM) that does not utilize tiling to exploit data locality in cache.

A hotspot caused by conditional execution (e.g., if statements) within a parallelized loop, forcing threads in the same warp or wavefront to serialize.

• Symptoms: Severe underutilization of parallel lanes on the NPU's SIMD units.
• Identification: Profiling occupancy metrics and analyzing execution traces for branches within data-parallel kernels.
• Example: Implementing a ReLU activation max(0, x) where some threads take the 'then' path and others the 'else' path, causing pipeline stalls.

Hotspots arising from non-coalesced memory accesses or suboptimal data structures that prevent efficient use of the memory hierarchy.

• Symptoms: Low effective memory bandwidth, high latency for memory operations.
• Identification: Memory coalescing analysis in profilers and inspecting cache hit rate metrics.
• Example: Accessing a column of a row-major matrix in a tight loop, resulting in scattered, uncoalesced loads that cannot be batched into a single wide transaction.

A systemic hotspot where the cost of launching many small, fine-grained kernels dominates the total execution time, often due to excessive host-device synchronization.

• Symptoms: High host CPU usage by the driver, poor overall NPU utilization despite fast individual kernels.
• Identification: Timeline traces in profilers (e.g., NVIDIA Nsight Systems, AMD ROCprof) showing many short kernels separated by gaps.
• Example: A loop on the host CPU that launches a tiny NPU kernel for each element or small batch, instead of using batched execution or fusing operations.

Operations that perform minimal computation on each data element loaded from memory, making them fundamentally limited by memory bandwidth.

• Symptoms: Performance is identical regardless of using FP32 or lower precision (FP16/INT8), indicating the bottleneck is not compute.
• Identification: Performance models or roofline analysis showing the kernel's operational intensity falls in the memory-bound region.
• Example: Adding a bias vector to a tensor or applying a simple scaling factor. The solution is often kernel fusion with a preceding compute-heavy operation.

A hotspot that emerges not from the kernel itself, but from sharing hardware resources with other concurrent workloads, leading to unpredictable performance degradation.

• Symptoms: High variance in kernel execution times, reduced memory bandwidth or cache hit rate when other jobs are running.
• Identification: System-wide profiling and monitoring of shared resources (L2 cache, memory controllers).
• Example: Two different AI models running concurrently on the same NPU, competing for L2 cache space and causing thrashing.

A hardware register within a processor that counts low-level architectural events. These counters provide the raw data for hotspot analysis by quantifying events like:

• Cache misses (L1, L2, L3)
• Floating-point operations (FLOPS)
• Branch mispredictions
• Memory transactions
• Instruction retirements Profiling tools sample these counters to attribute costs to specific functions or lines of code, moving from high-level timing to microarchitectural understanding.

The systematic process of identifying the primary limiting factor that restricts the overall performance of a system or computational kernel. The core question is: What is the system waiting for?

Common bottlenecks in NPU workloads include:

• Compute-Bound: Limited by ALU throughput; the processor is saturated with arithmetic operations.
• Memory-Bound: Limited by memory bandwidth or latency; compute units idle waiting for data.
• Latency-Bound: Limited by dependency chains or serial operations.
• Contention-Bound: Limited by shared resource conflicts (e.g., cache, memory ports). Analysis uses ratios of performance counter values (e.g., FLOPS/byte) to classify the bottleneck.

A visualization tool for hierarchical profiler data, invented by Brendan Gregg. It provides an intuitive, top-down view of where CPU or NPU time is spent across the entire call stack.

Key characteristics:

• Each horizontal bar represents a function in the call stack.
• Bar width is proportional to the function's presence in the samples (time or count).
• The y-axis shows stack depth.
• Colors are typically random for differentiation.

For hotspot identification, flame graphs allow engineers to quickly see wide functions that consume significant time, as well as deep call chains. Interactive versions let you click to zoom into specific code paths.

A profiling method that periodically interrupts program execution (e.g., via a timer or hardware performance interrupt) to record the current program counter and call stack. It constructs a statistical profile of where time is spent.

Advantages for Hotspot Identification:

• Low Overhead: Typically 1-5%, suitable for production profiling.
• System-Wide View: Can capture time spent in the kernel, libraries, and the application.
• Statistical Significance: With enough samples, it accurately reveals the most frequent hotspots.

Limitations:

• Can miss short-lived functions if the sampling rate is too low.
• Provides statistical estimates, not exact instruction counts. Tools like perf on Linux and VTune use this method.

A profiling method that inserts measurement code (instrumentation) into the target program at compile time or runtime. This allows for precise, deterministic measurement of specific code regions.

Common Techniques:

• Source Instrumentation: Adding timing calls manually or via preprocessor macros.
• Compiler Instrumentation: Using flags like -pg for gprof to inject counting code.
• Binary Instrumentation: Tools like DynamoRIO or PIN rewrite the binary to add probes.

Use Case in Hotspot ID: Provides exact counts of function calls and precise elapsed time for small, critical sections where sampling profiler granularity is insufficient. The trade-off is significantly higher overhead, which can distort the behavior of the system being measured.

A chronological, detailed record of the dynamic sequence of operations performed during a program's run. Unlike a profile which aggregates data, a trace is a timeline of events.

Traced events can include:

• Function entries and exits
• Memory allocations and accesses
• System calls
• Kernel launches and completions (for NPU/GPU)
• Inter-thread communications

Value for Hotspot Analysis: Traces are essential for understanding temporal relationships and causality. They can reveal patterns like:

• Irregular pauses between compute kernels (synchronization issues).
• The exact sequence leading to a cache miss.
• Dependencies causing pipeline stalls. Tools like LTTng, NVIDIA Nsight Systems, and Intel Trace Analyzer generate and visualize execution traces.