Flame Graph

A flame graph visualizes the call stack from bottom to top. The root of the stack (the main thread or entry point) is at the bottom. Each frame (function) is a horizontal bar. A bar's width is proportional to the time spent in that function, including all its descendants. This hierarchy allows you to see not just which function is slow, but the entire expensive code path leading to it.

• Bottom-up vs. Top-down: The standard (icicle) flame graph is top-down, showing the caller above the callee. A reverse flame graph (or bottom-up) inverts this to aggregate costs by leaf functions, useful for identifying common low-level resource consumers.

The fundamental metric in a flame graph is the horizontal width of each frame. This width is not absolute time but a relative proportion of the total sampled time. This encoding allows for immediate visual comparison: the widest bars represent the hottest code paths and the most significant optimization targets.

• Sampling Overhead: Because flame graphs are typically generated from sampling profilers (e.g., perf, DTrace), they have low overhead and provide a statistical profile of where the program spends its cycles, rather than exact instruction counts.

Color is used as a non-quantitative visual aid to differentiate between frames or to encode an additional dimension. Common schemes include:

• Random Hue: Different colors for adjacent frames to improve visual separation.
• Library/Namespace: Color by code origin (e.g., user code vs. kernel code vs. library code).
• Performance Metric: In differential flame graphs, color can show performance regression (red) or improvement (green) between two profiles.

Color should never be the primary carrier of the performance data; that role is reserved for width.

Static flame graphs are useful, but interactive exploration is key for deep analysis. Interactive features include:

• Zooming: Click on a frame to zoom into that subtree, hiding all other call paths.
• Search: Highlight all frames matching a function name, library, or pattern.
• Tooltips: Hover over a frame to see exact sample counts, percentages, and source file locations.

This interactivity transforms the graph from a static report into a diagnostic tool, allowing engineers to drill down from a broad system-wide view to a specific, costly loop or system call.

While CPU time is the most common profile, the flame graph visualization can be applied to many other performance resources:

• CPU Flame Graph: The original type, showing on-CPU function time.
• Off-CPU Flame Graph: Shows time a thread was blocked (e.g., on I/O, locks, sleeping), revealing latency issues.
• Memory Flame Graph: Visualizes heap allocations or memory footprint by call site.
• Differential Flame Graph: Compares two profiles, showing which code paths increased (red) or decreased (green) in cost.
• Hot/Cold Flame Graph: Combines on-CPU and off-CPU data in a single visualization.

For Neural Processing Unit (NPU) acceleration, flame graphs can be generated from hardware execution traces and kernel profiler data. Instead of software function calls, frames represent:

• NPU Kernels: Individual computational kernels launched.
• Memory Operations: Data transfers (H2D, D2H) and synchronization events.
• Hardware Stages: Pipeline stages, tensor operations, or vendor-specific low-level functions.

This provides a visual breakdown of the kernel execution timeline, helping identify if bottlenecks are in compute-bound kernels, memory-bound transfers, or scheduling overhead between concurrent kernels.

A hardware register within a processor that counts low-level architectural events. On NPUs, these counters are crucial for moving beyond timing data to understand why a kernel is slow. Common counters include:

• Cache Misses: Counts of L1/L2 cache access failures.
• FLOPs: Floating-point operations completed.
• Memory Transactions: Volumes of data moved between hierarchy levels.
• Stall Cycles: Cycles where execution pipelines are idle. Flame graphs can be annotated with counter data to show not just where time is spent, but which hardware bottlenecks are present.

The process of using profiling data to locate the specific functions, loops, or instructions within a program that consume the most execution time or resources. A flame graph is the quintessential hotspot identification tool because:

• Width is proportional to time or resource consumption.
• Hierarchical stack context shows the calling path to the hotspot.
• Visual pattern matching allows engineers to quickly spot the widest bars, which represent the most significant optimization targets.

The systematic process of determining the primary limiting factor that restricts the overall performance of a system or kernel. Flame graphs help diagnose common bottlenecks:

• Compute-Bound: Wide bars in pure computation functions (e.g., matrix multiplications).
• Memory-Bound: Wide bars in memory copy routines or functions waiting on data.
• Synchronization-Bound: Wide bars in lock or barrier functions. This analysis moves from where time is spent (the hotspot) to why it is spent there, guiding the selection of appropriate optimization strategies.

A chronological, detailed record of the sequence of instructions, memory accesses, and function calls executed by a program. While a flame graph shows aggregated statistical samples, an execution trace provides a deterministic, step-by-step replay of execution. Key differences:

• Flame Graph: Statistical, aggregated, low-overhead. Answers "Where is the time spent?"
• Execution Trace: Deterministic, sequential, high-overhead. Answers "What was the exact order of events?" Traces are used for debugging correctness, while flame graphs are for performance profiling.

A profiling method that periodically interrupts program execution (e.g., via a timer or hardware event) to record the current program counter and call stack. This statistical approach is the primary data collection method for CPU flame graphs. Characteristics include:

• Low Overhead: Typically 1-2% slowdown.
• Statistical Accuracy: Provides a representative picture of where time is spent.
• System-Wide: Can profile the entire system, not just a single application. The sequence of collected stack samples is then aggregated and visualized to create the flame graph.