Distributed Tracing

A span is the fundamental unit of work in a distributed trace. It represents a single, named, and timed operation representing a piece of the workflow, such as:

• A function call or internal computation.
• An HTTP request to an external service.
• A database query or cache operation.

Each span contains a unique ID, a parent span ID (to establish hierarchy), a name, start/end timestamps, and a set of key-value attributes (tags) and events (annotated logs). Spans are the building blocks from which a complete trace is assembled.

A trace is a directed acyclic graph (DAG) of spans that represents the complete end-to-end path of a request through a distributed system. It is defined by a single, unique Trace ID that is shared by all spans belonging to that request. A trace visualizes:

• The causal and temporal relationships between operations.
• Parallel and sequential execution paths.
• The aggregate latency of the entire transaction.

Traces provide the holistic view necessary to diagnose performance issues that span multiple services, answering the critical question: 'What happened to my request?'

Trace context propagation is the mechanism that carries essential identifiers across service boundaries, enabling the correlation of spans into a coherent trace. This context, containing the Trace ID and the current Span ID, is typically injected into:

• HTTP headers (using standards like W3C TraceContext).
• gRPC metadata.
• Messaging payloads (e.g., Kafka, RabbitMQ headers).
• In-process context (e.g., thread-local storage).

Without reliable propagation, each service would create isolated, unrelated spans, breaking the end-to-end visibility that defines distributed tracing.

Attributes (also called tags) are key-value pairs attached to a span that provide descriptive, queryable metadata about the operation. Examples include:

• http.method="GET", http.status_code=200
• db.system="postgresql", db.statement
• agent.decision="retry", user.id="abc123"

Events are timestamped annotations on a span that record discrete occurrences during its lifetime, such as:

• exception with stack trace.
• A log message indicating a state change.
• agent.reflection marking a reasoning step.

These enrich spans with the context needed for effective debugging and analysis.

Instrumentation is the code added to an application to generate spans and propagate context. It exists in two primary forms:

• Manual Instrumentation: Developers explicitly write code to create spans, add attributes, and handle context propagation using an SDK (e.g., OpenTelemetry). This offers maximum control and customization.
• Automatic Instrumentation: Language-specific agents or libraries automatically inject tracing code at runtime for common frameworks (HTTP clients/servers, database drivers, messaging libraries). This provides immediate observability with minimal code changes.

Effective tracing requires strategic instrumentation at all critical integration points within and between services.

Sampling is a critical strategy for controlling the volume and cost of trace data by selectively deciding which requests to trace. The two primary approaches are:

• Head-based Sampling: The sampling decision (e.g., 'record this trace') is made at the very start of a request (e.g., by the first service or load balancer) and is propagated and enforced downstream. This is simple but may discard interesting traces that only become problematic later.
• Tail-based Sampling: The decision is made after a trace is complete, based on its aggregated properties (e.g., total duration, error status, presence of specific attributes). This allows capturing all error traces or slow traces but requires a buffering component (like the OTel Collector) to hold spans before making the decision.

A span is the fundamental building block of a distributed trace, representing a single, named, and timed operation representing a unit of work. It contains metadata such as:

• Operation name (e.g., POST /api/agent/plan)
• Start and end timestamps
• Key-value attributes (e.g., agent.id="agent-7f2a", tool.name="search_database")
• Status code (OK, ERROR)
• Events and links to other spans Spans are nested to form parent-child relationships, creating a hierarchical view of a request's journey. For an autonomous agent, a single planning cycle might be a parent span, with child spans for tool execution, LLM API calls, and memory retrievals.

Trace context is the metadata that propagates the identity of a distributed trace across service and process boundaries. It contains the essential identifiers needed to correlate all spans belonging to the same logical request. The primary components are:

• Trace ID: A globally unique 16-byte identifier for the entire request flow.
• Span ID: A unique 8-byte identifier for the current unit of work.
• Trace flags: Control flags, most importantly the sampling decision.
• Trace state: Provides additional vendor-specific propagation data. This context is typically injected into and extracted from transport headers (HTTP, gRPC, message queues) using standards like W3C TraceContext, ensuring interoperability between different services, programming languages, and observability vendors.

Tail-based sampling is an intelligent sampling strategy where the decision to keep or discard a complete trace is made after the request has finished, based on its aggregated properties. This contrasts with head-based sampling, which decides at the start of a request. A telemetry pipeline using tail-based sampling will:

1. Collect 100% of spans for a short buffer period.
2. After the trace is complete, evaluate it against predefined rules.
3. Only export traces that match criteria like:
   • Contains an error (status=ERROR)
   • Duration exceeds a threshold (e.g., > 5 seconds)
   • Involves a specific service or agent
   • Matches a custom attribute (e.g., user.tier="premium") This method is highly effective for agent observability, as it guarantees visibility into all anomalous or slow executions—precisely the traces most valuable for debugging—while dramatically reducing storage costs by discarding routine, successful traces.