Trace

The Root Span is the initial and top-most span in a trace, representing the entry point of the entire operation, such as an agent receiving a user query. It establishes the Trace ID and initial timing context. All other spans in the trace are its children.

• Purpose: Defines the trace's temporal boundaries and overall success/failure state.
• Example: A span named /agent/process with a duration covering the agent's entire task lifecycle.

Child Spans are nested operations that occur within the context of a parent span, representing sub-steps like individual tool calls, LLM invocations, or database queries. They inherit the parent's Trace ID and have their own Span ID and timing.

• Hierarchy: Forms a tree structure, enabling detailed breakdowns of complex workflows.
• Causality: The parent-child relationship explicitly shows which operation called another.
• Example: Under a root process_task span, child spans for call_weather_api, generate_response, and update_log.

Trace Context Propagation is the mechanism that carries the Trace ID and active Span ID across process and network boundaries (e.g., via HTTP headers like traceparent). This is critical for Distributed Tracing, allowing spans from different services—including external APIs—to be linked into a single coherent trace.

• Standard: Often implemented using the W3C Trace Context standard.
• Agentic Use: Enables tracking an agent's request as it flows from the orchestrator, to an LLM, to an external tool, and back.

Span Attributes are key-value pairs that annotate a span with descriptive metadata (e.g., tool.name="google_search", http.status_code=200). Span Events are timestamped logs attached to a span, marking significant occurrences (e.g., exception.thrown, cache.hit).

Together, they provide the forensic details needed to understand what happened during an operation:

• Attributes for State: user.id, agent.session, request.parameters.
• Events for Moments: retry.attempted, function.entered, decision.made.

A Flame Graph is the primary visualization for a trace, displaying spans as horizontal bars stacked vertically by parent-child relationship. The width of each bar represents the span's duration.

• Performance Analysis: Instantly identifies the critical path (the longest chain of dependencies) and latency bottlenecks.
• Debugging: Color-coding by service or error status highlights problematic operations.
• Tool Call Insight: Clearly shows serial vs. parallel tool execution, blocking calls, and the proportion of time spent waiting on external APIs.

Aggregating data from many traces generates Trace-Based Metrics, which provide system-wide performance and reliability insights. These are derived from span attributes and timing data.

Key derived metrics for agentic systems include:

• P95 Latency: The 95th percentile of total trace duration.
• Error Rate: Percentage of traces containing a span with an error status.
• Service Dependency Map: Automatically generated by analyzing which services call others across all traces.
• Cost Attribution: Summing token counts or API call costs from spans tagged with a cost_center attribute.

A Span is the fundamental building block of a trace, representing a single, named, and timed operation within the larger workflow. In agentic observability, each discrete action—such as a tool call, an LLM inference step, or a database query—is captured as a span.

• Structure: Contains an operation name, start/end timestamps, status code, and a unique ID.
• Hierarchy: Spans have parent-child relationships, forming the tree structure of a trace.
• Example: A single API call to process_payment or a call to get_weather would each be a distinct span within an agent's task trace.

Distributed Tracing is the methodology and infrastructure for following a request—like an agent's task—as it propagates across service boundaries, including external APIs and internal microservices. It solves the challenge of observability in complex, distributed systems.

• Core Mechanism: Uses a trace ID propagated via headers (e.g., traceparent) to link spans from different services.
• Value: Provides a unified, end-to-end view of performance and failure points across an agent's entire execution graph, from initial prompt to final action.

Span Attributes are key-value pairs attached to a span that provide descriptive, queryable metadata about the operation. They turn raw timing data into rich, contextual information for debugging and analysis.

• Examples for Tool Calls:
  • tool.name: "stripe_charge_api"
  • http.status_code: 429
  • `agent.session_id": "sess_abc123"
  • `llm.model": "gpt-4-turbo"
• Use Case: Enables filtering and grouping traces, e.g., "show all traces where tool.name equals 'send_email' and http.status_code is 500."

Trace Correlation is the technical process of ensuring all telemetry signals generated during a single logical execution are linked together via a shared identifier. It is what binds disparate spans into a coherent trace.

• Primary Identifier: The Trace ID is generated at the start of a request and must be passed along with every subsequent call.
• Propagation: Typically implemented using standardized headers like W3C's traceparent or B3 headers.
• Critical for Agents: When an agent calls multiple external tools, correlation ensures the tool provider's spans (if instrumented) can be linked back to the agent's originating trace.

A Service Level Indicator (SLI) is a quantitative measure of a service's performance or reliability from the user's (or agent's) perspective. For tool call instrumentation, SLIs are derived from trace and span data.

• Common Agentic SLIs:
  • Latency: P95 tool call response time.
  • Success Rate: Percentage of tool calls that complete successfully.
  • Availability: Percentage of time the tool/API is reachable.
• Foundation for SLOs: SLIs are the raw metrics used to define Service Level Objectives (SLOs), which are target thresholds for reliability.

The Circuit Breaker Pattern is a resilience design pattern that prevents an agent from repeatedly calling a failing tool or service. It monitors failure rates (often via trace/span error data) and opens the circuit to fail fast, allowing the downstream service time to recover.

• Three States:
  • Closed: Requests flow normally (tool is healthy).
  • Open: Requests fail immediately without calling the tool (tool is unhealthy).
  • Half-Open: A limited number of test requests are allowed to probe for recovery.
• Observability Integration: The opening/closing of the circuit should be emitted as Span Events within traces, providing clear causality for failures.