Flame Graph

A flame graph represents the call stack of a program or trace as a set of nested, horizontal rectangles. Each rectangle, or frame, represents a function or span. The vertical axis shows stack depth, with the root span at the bottom and child spans stacked above it. This nesting directly maps to the parent-child relationships defined by span IDs within a trace, making the execution flow immediately apparent.

The primary quantitative insight comes from the width of each frame. In a CPU profile flame graph, width is proportional to the time spent in that function. In a distributed tracing context, width typically represents the duration of a span. This allows engineers to visually identify hot code paths or latency bottlenecks at a glance—the widest frames consume the most resources. The graph aggregates samples, so width represents the sum of all invocations of that function/span.

Color is used as a consistent, non-quantitative visual aid to improve readability and differentiate between types of operations. Common schemes include:

• Hue by library or namespace (e.g., green for application code, red for database calls, blue for HTTP clients).
• Saturation by resource type (e.g., different shades for CPU vs. I/O waits).
• Monochromatic to reduce cognitive load, where color simply helps distinguish adjacent frames. Color does not encode magnitude; the width carries all quantitative information.

Modern flame graph implementations are interactive visualizations. Key interactions include:

• Click-to-zoom: Clicking a frame zooms the view to show only that stack and its children, enabling detailed inspection of deep call paths.
• Search highlighting: Searching for a function or service name highlights all matching frames across the graph.
• Tooltip details: Hovering over a frame reveals precise metadata, such as span name, duration, span attributes, and percentage of total trace time. This interactivity transforms a static profile into an investigative tool for performance debugging.

A flame graph is an aggregated visualization. It does not show every individual function call or span instance in a timeline. Instead, it merges all sampled stack traces or spans, summing their durations. This aggregation is powerful for identifying statistically significant bottlenecks across many requests. For example, if a specific database query appears wide, it indicates that query is a major contributor to latency across the sampled traces, not just in one anomalous request.

When applied to distributed tracing, a flame graph visualizes a single trace or an aggregate of traces. Each frame corresponds to a span. The hierarchy shows the propagation of work across services. This provides a unified view of end-to-end latency, revealing whether time is spent in a specific microservice, a particular tool call, or in network communication between spans. It bridges the gap between traditional profiling and distributed systems observability, making complex trace data intuitively scannable.

A flame graph derived from a distributed trace reveals the service topology and call hierarchy for a specific request. It shows how work propagates from a root span through various downstream services.

• Visualizing fan-out: See parallel calls to multiple services and identify if one slow dependency is serializing the entire workflow.
• Mapping code-to-infrastructure: Connect business logic (function names in spans) to the underlying infrastructure components (database, cache, external APIs) they invoke.

The horizontal stacking in a flame graph clearly distinguishes sequential operations (stacks of bars) from concurrent operations (bars side-by-side at the same depth). This is critical for optimizing asynchronous workflows.

• Identifying blocking calls: Spot where the execution could be parallelized but is currently sequential, creating artificial latency.
• Validating async patterns: Confirm that intended concurrent operations (e.g., fan-out API calls) are executing in parallel as designed.

By mapping time spent to specific functions and services, flame graphs enable granular cost attribution. In agentic systems, this is essential for understanding the compute cost of specific reasoning steps or tool calls.

• Token usage correlation: In LLM-based agents, correlate wide bars (long durations) with high-token-count prompts or completions.
• External API cost analysis: Identify which third-party tool calls are the most expensive in terms of both latency and direct API costs.

For autonomous agents, a flame graph visualizes the planning, execution, and reflection cycle. Each major loop iteration appears as a distinct set of frames, allowing engineers to see time spent in cognitive phases versus tool execution.

• Inefficient planning: Identify agents stuck in excessive planning or reflection, indicated by deep, wide stacks of LLM calls.
• Tool execution profiling: See the exact sequence and duration of external tool calls (API, database, code execution) within an agent's action phase.

Flame graphs serve as a visual baseline for normal performance. Automated systems can diff flame graph shapes or aggregate span durations across deployments to detect regressions.

• Post-deployment analysis: Compare aggregate flame graphs from before and after a code deploy to see if new spans were added or existing ones became slower.
• Anomaly detection: Flag traces where the flame graph shape deviates significantly from the norm, indicating potential performance anomalies or errors.

A span is the fundamental unit of work in distributed tracing, representing a named, timed operation for a contiguous segment of work within a service. In a flame graph, each horizontal rectangle (or "flame") corresponds to a span.

• Key Properties: Contains a start/end timestamp, a name, span attributes (key-value metadata), and a span kind (e.g., Client, Server).
• Hierarchy: Spans have parent-child relationships, forming the nested stack visualized in a flame graph. The width of a span's rectangle represents its duration.

A trace is a collection of spans that represents the complete end-to-end path of a single request as it propagates through a distributed system. A flame graph visualizes one entire trace.

• Structure: Spans within a trace form a directed acyclic graph (DAG), though flame graphs typically show a simplified, aggregated call stack view.
• Correlation: All spans in a trace share a unique Trace ID, enabling trace correlation with logs and metrics for unified debugging.

Distributed tracing is the overarching methodology of instrumenting applications to observe requests as they flow across service boundaries. Flame graphs are a primary diagnostic output of this practice.

• Purpose: Used to understand system latency, diagnose performance bottlenecks, and visualize service dependencies.
• Mechanism: Relies on distributed context propagation (e.g., via W3C Trace Context headers) to pass trace IDs and span IDs between services, maintaining continuity.

OpenTelemetry (OTel) is the vendor-neutral, open-source standard for generating, collecting, and exporting telemetry data, including traces. It is the primary source of data for modern flame graphs.

• Components: Includes APIs/SDKs for instrumentation, the OTLP protocol for data export, and the OpenTelemetry Collector for processing.
• Role: Provides the standardized span and trace data model that visualization tools like flame graphs consume. Auto-instrumentation via OTel agents is a common way to generate this data without code changes.

A service graph is a complementary visualization to a flame graph. While a flame graph shows the internal call stack of a single request, a service graph shows the macro-level dependencies between services across all requests.

• Derivation: Automatically generated by aggregating span data from many traces to identify which services call each other.
• Use Case: Used for architectural understanding, identifying upstream/downstream impacts of failures, and validating deployment topology.

Tail sampling is a critical strategy for managing trace data volume before visualization. It decides whether to keep or discard a trace after the request is complete, based on its full context.

• Contrast with Head Sampling: Head sampling decides at the request's start, potentially missing interesting traces that only exhibit problems (e.g., high latency) later.
• Flame Graph Relevance: Enables cost-effective storage by only retaining traces that are most valuable for analysis, such as those with errors or exceeding latency thresholds, which are prime candidates for flame graph inspection.