Kernel Profiler

The profiler measures the wall-clock time a kernel takes to complete, from launch to finish. This includes:

• Kernel Launch Overhead: The latency between the host CPU issuing the command and the accelerator beginning execution.
• Pure Compute Time: The time spent actively processing data on the NPU's compute units.
• Host-Device Synchronization: Time spent waiting for the kernel to complete before the host can proceed.

This is the primary metric for determining if an optimization is successful, measured in microseconds (µs) or milliseconds (ms).

This metric quantifies the raw computational power utilized, measuring the rate of arithmetic operations performed. It's critical for determining if a kernel is compute-bound.

• FLOPS: Floating-Point Operations Per Second. Measured for kernels using FP32, FP16, or BF16 data types.
• TOPS: Tera-Operations Per Second. Often used for integer operations (e.g., INT8) common in quantized models.

The profiler calculates this by correlating the known operation count of the kernel (e.g., from the model graph) with the measured execution time. Achieving a high percentage of the NPU's theoretical peak FLOPS/TOPS is a key optimization goal.

Measures the effective rate of data transfer between the NPU's compute cores and its memory hierarchy (e.g., global/HBM memory, shared/L1 cache). It identifies memory-bound kernels.

• Bytes Read/Written: The total volume of data moved during kernel execution.
• Achieved Bandwidth: Calculated as (Bytes Transferred) / (Kernel Time), measured in GB/s.
• Bandwidth Saturation: The ratio of achieved bandwidth to the hardware's theoretical peak memory bandwidth. Low saturation indicates poor memory access patterns or insufficient computational intensity to hide latency.

Low-level hardware registers that count specific microarchitectural events, providing deep insight into efficiency. Common counters include:

• Cache Hit/Miss Rates: For L1, L2, or shared memory caches. A low hit rate indicates poor data locality.
• DRAM Accesses: Counts of reads and writes to main memory.
• Compute Unit Utilization: Measures the activity rate of ALUs, vector units, or tensor cores.
• Stall Reasons: Counts cycles where the pipeline is stalled due to memory dependency, instruction fetch, or synchronization.

Analyzing these counters pinpoints the exact microarchitectural bottleneck, such as memory contention or low instruction-level parallelism.

Measures how effectively the kernel utilizes the NPU's parallel execution resources.

• Occupancy: The ratio of active warps/wavefronts (thread groups) to the maximum supported per streaming multiprocessor (SM). High occupancy helps hide memory latency.
• Thread Divergence: Tracks execution paths where threads within a warp take different branches, causing serialization and reduced efficiency.
• Vectorization Efficiency: Assesses how well the kernel's instructions utilize the processor's SIMD (Single Instruction, Multiple Data) or vector units.

Low occupancy or high divergence indicates suboptimal workgroup sizing or control flow, leaving hardware resources idle.

For edge and mobile deployments, profiling power draw is essential. This metric correlates performance with efficiency.

• Average Power (Watts): The mean electrical power consumed by the NPU during kernel execution.
• Energy (Joules): The total energy used, calculated as (Average Power) × (Execution Time).
• Performance-per-Watt: A key efficiency metric, often expressed as FLOPS/Watt or inferences-per-second per watt.

Profiling this helps optimize for battery life in edge devices and reduce operational costs in data centers by identifying kernels with disproportionate energy costs.

A hardware register within a processor that counts low-level microarchitectural events. Kernel profilers read these counters to quantify behavior that directly impacts performance.

• Examples: Cache misses, floating-point operations (FLOPs), branch mispredictions, memory transactions.
• Purpose: Provides objective, hardware-verified data on why a kernel is slow (e.g., high L1 cache miss rate).
• Use Case: Correlating high kernel latency with a specific bottleneck like memory bandwidth saturation.

A chronological, detailed log of a program's execution at the instruction or API level. While a profiler provides statistical summaries, a trace offers a sequential replay of events.

• Granularity: Can record individual kernel launches, memory copies, CUDA/HIP API calls, and even instruction streams.
• Analysis: Used for understanding dependencies, identifying idle gaps between kernels, and debugging correctness issues.
• Tool Example: NVIDIA Nsight Systems, AMD ROCm ROCprofiler (in tracing mode), Intel VTune.

The analytical process of using profiler data to locate the specific code segments consuming the most resources. It is the primary action taken after data collection.

• Method: Profilers rank functions or source lines by metrics like exclusive time (time spent in the function itself) or inclusive time (time including sub-calls).
• Output: A ranked list or visualization (e.g., a Flame Graph) pointing engineers to the most critical optimization targets.
• Goal: Direct optimization effort to the code sections that will yield the largest performance return.

The diagnostic process of determining the fundamental limiting factor constraining a kernel's performance. It moves beyond identifying where time is spent to explain why.

• Common Bottlenecks:
  • Compute-Bound: Limited by ALU throughput (FLOPS).
  • Memory-Bound: Limited by memory bandwidth or latency.
  • Latency-Bound: Limited by instruction or memory dependency chains.
• Profiler Role: Uses metrics like compute throughput, memory bandwidth utilization, and occupancy to classify the bottleneck.

A profiling methodology that periodically interrupts the processor to record the current program counter and call stack. It provides a statistical, low-overhead view of performance.

• Mechanism: Uses a timer or hardware performance interrupt to 'sample' execution state.
• Advantage: Very low intrusion, suitable for profiling live, long-running applications.
• Limitation: Can miss short-duration, high-frequency events unless sampling rate is extremely high.

A profiling methodology that inserts data collection code directly into the target program. This provides precise, event-driven measurements but adds overhead.

• Mechanism: Source code or binary is modified to call profiling libraries at function entry/exit or around specific code blocks.
• Advantage: High precision for timing specific regions; captures every invocation.
• Use Case: Detailed analysis of small, critical kernels where overhead is acceptable or can be subtracted.
• Contrast: Compared to Sampling Profilers, instrumentation is more intrusive but more accurate for targeted regions.