Trace

A span is the fundamental building block of a trace, representing a single, named, and timed operation within a service. It captures the execution of a contiguous unit of work, such as:

• A function call or method execution
• An HTTP request to an external API
• A database query or transaction

Each span contains a start time, duration, status code (e.g., OK, ERROR), and a set of attributes for metadata. Spans are linked via parent-child relationships to form the trace's hierarchical structure.

Unique identifiers are crucial for correlating data across a distributed system.

• Trace ID: A globally unique, immutable identifier (typically a 16-byte array) assigned to the entire request. Every span in the same trace shares this ID, enabling end-to-end correlation.
• Span ID: A unique identifier for an individual span within a trace. It is used to establish the parent-child links between spans.
• Parent Span ID: The identifier of the span that directly caused the current span's work. A span without a parent ID is a root span, representing the initial operation of the trace.

Span context is the immutable tracing state that must be propagated across process and service boundaries to maintain trace continuity. It contains the trace ID, span ID, trace flags (e.g., sampling decision), and trace state (vendor-specific data).

Distributed context propagation is the mechanism for passing this context, typically via:

• HTTP headers (using standards like W3C Trace Context or B3 Propagation)
• Messaging system metadata (e.g., Kafka headers, gRPC metadata)

A propagator component in the tracing SDK handles the injection (outbound) and extraction (inbound) of this context.

These components add rich, queryable metadata to a span.

• Attributes: Key-value pairs that describe the operation. Examples include http.method="GET", db.statement="SELECT * FROM users", or custom business data like user.id="12345".
• Events: Timed, structured logs attached to a span that represent singular occurrences during its lifetime, such as an exception being thrown, a cache miss, or a significant state change. Each event has a name, timestamp, and its own set of attributes.

These elements define semantic relationships and roles.

• Span Links: A reference from one span to a span in a different trace. They model causal relationships that are not parent-child, such as a batch job processing an item that originated from an asynchronous queue.
• Span Kind: A classification specifying the span's role in the trace topology. Core kinds include:
  • Server: For the receiver of a remote operation (e.g., an HTTP server handler).
  • Client: For the initiator of a remote operation (e.g., an outgoing HTTP call).
  • Internal: For operations within the application boundary with no remote context.
  • Producer/Consumer: For messaging systems.

A complete trace is a collection of spans that forms a Directed Acyclic Graph (DAG), not merely a linear chain. This structure emerges because:

• A single parent span can have multiple concurrent child spans (e.g., fan-out API calls).
• Span links can create edges between spans in different traces.
• The DAG structure is visualized in tools via flame graphs (showing nested duration) and service graphs (showing inter-service dependencies). The root span is the graph's entry point, and the collective timing of all spans defines the request's total latency.

A trace is a collection of spans that represents the end-to-end path of a single request or agent execution as it propagates through a distributed system. It forms a directed acyclic graph (DAG) where:

• Each span is a named, timed operation representing a unit of work.
• Parent-child relationships define the flow and hierarchy of operations.
• The root span initiates the trace, with subsequent spans as children or follows-from links. This structure is essential for visualizing the complete lifecycle of an agent's task, from initial prompt to final action.

A span is the atomic unit of a trace. For agentic observability, spans capture distinct phases of agentic work:

• Internal Reasoning: A span for a planning cycle, chain-of-thought, or reflection step.
• Tool/API Execution: A span for an external function call, database query, or API request, including duration and success status.
• Memory Operations: Spans for reading from or writing to a vector store or knowledge graph. Each span contains critical metadata: a span ID, parent span ID, start/end timestamps, span kind (e.g., INTERNAL, CLIENT), and attributes (key-value pairs detailing the operation).

For a trace to be truly distributed, span context must propagate across process and network boundaries. This is the mechanism that ties an agent's internal reasoning to its external API calls.

• Trace ID & Span ID: A globally unique Trace ID identifies the entire execution. The current Span ID identifies the specific operation.
• Propagation Standards: Context is carried via headers using standards like W3C Trace Context or B3 Propagation.
• Agentic Specifics: When an agent calls a tool, the SDK injects the current span context into the HTTP request. The tool's service extracts it, creating a child span, thereby extending the trace into the external service.

Raw spans are useful for timing; enriched spans are critical for auditing and debugging agent behavior. Trace enrichment adds semantic, business, and agent-specific context.

• Agent State: Attach the current goal, plan step, or conversation turn as span attributes.
• Tool Call Details: Enrich spans with the exact function name, parameters, and parsed results.
• Cost Telemetry: Add attributes for LLM token usage, model name, and API call cost.
• Business Context: Include user ID, session ID, or transaction ID to link technical traces to business outcomes.

Traces are visualized to diagnose performance and understand flow.

• Flame Graph: A hierarchical visualization where the width of a horizontal bar represents a span's duration. It instantly shows the critical path and which internal reasoning step or tool call caused latency.
• Service Graph (or Dependency Map): A topological map automatically generated from trace data. For multi-agent systems, it shows agents as nodes and their communication (RPC, messages) as edges, revealing the interaction network and upstream/downstream dependencies.

Capturing every trace is often prohibitively expensive. Trace sampling strategically reduces volume.

• Head Sampling: Decision made at the start of a request (e.g., sample 10% of all agent sessions). Simple but may miss rare, important traces.
• Tail Sampling: Decision made after the trace is complete, based on its full content. Crucial for agents, as it allows rules like:
  • Sample 100% of traces with errors (e.g., tool call failure).
  • Sample 100% of traces where final answer confidence < threshold.
  • Sample traces with latency > 2s. This ensures high-value agent executions are always retained for analysis.

A span is the fundamental building block of a trace, representing a single, named, and timed operation within a service. It captures the work done for a specific unit of logic, such as:

• A function call or method execution
• A database query
• An HTTP request to an external API

Each span contains a start time, duration, status code, and span attributes (key-value metadata). Spans are linked in parent-child relationships to form the hierarchical structure of a trace.

Distributed tracing is the overarching methodology for profiling and monitoring requests as they flow through a distributed system. It involves:

• Instrumenting services to emit spans.
• Propagating trace context (like a Trace ID) across service boundaries via HTTP headers or message metadata.
• Collecting and visualizing these correlated spans to reconstruct the complete request path.

The primary goal is to understand system performance, identify bottlenecks (visualized in a flame graph), and diagnose failures in microservices or agentic architectures.

OpenTelemetry (OTel) is the open-source, vendor-neutral standard for generating, collecting, and exporting telemetry data. It is the de facto framework for implementing distributed tracing. Key components include:

• APIs and SDKs for manual and auto-instrumentation.
• The OpenTelemetry Collector, a proxy for receiving, processing, and exporting data.
• OTLP (OpenTelemetry Protocol), the standard gRPC/HTTP protocol for data transmission.
• Standardized semantic conventions for span attributes (e.g., http.method, db.query).

Trace sampling is the critical process of selectively capturing a subset of traces to balance observability depth with data volume and cost. Two primary strategies are:

• Head Sampling: The sampling decision is made at the start of a request (e.g., 10% of all traces). It's efficient but may miss important late-breaking events.
• Tail Sampling: The decision is made after the trace is complete, based on its full context (e.g., "sample all traces with latency > 2s or an error status"). This is more powerful but requires buffering traces in a collector.

Sampling rules are essential for production-scale observability.

W3C Trace Context is a formal W3C recommendation that defines a standard HTTP header format (traceparent, tracestate) for distributed context propagation. It ensures interoperability between different tracing systems and libraries by providing a unified way to carry:

• The Trace ID and Span ID.
• Sampling flags.
• Vendor-specific tracing state.

This standard replaced earlier, vendor-specific formats like B3 Propagation, enabling seamless tracing across heterogeneous technology stacks.

A service graph is a dynamic, topological map automatically derived from aggregated trace data. It visualizes the services (nodes) and the request dependencies or calls (edges) between them. This provides a system-wide view for:

• Identifying unexpected or brittle dependencies.
• Understanding the critical path of requests.
• Detecting changes in the communication topology after deployments.

Unlike a single trace, which shows one request's path, a service graph synthesizes data from many traces to show the architectural relationships and health of the entire ecosystem.