Distributed Tracing

A trace is the overarching container that represents the entire end-to-end journey of a single request or transaction as it propagates through a distributed system. It is uniquely identified by a Trace ID, a 128-bit or 64-bit random number generated at the very start of the request. All operations spawned by that initial request share this same Trace ID, allowing them to be correlated. A trace is conceptually a directed acyclic graph (DAG) of spans, where the edges represent causal relationships (parent-child links).

A span represents a single, named, and timed operation within a trace. It is the fundamental building block. Each span encapsulates a unit of work, such as:

• A service call (e.g., checkout-service.process)
• A database query
• An external API call

A span contains:

• Span ID: A unique identifier for this specific operation.
• Parent Span ID: The ID of the span that caused this work to happen (except for the root span). This establishes causality.
• Name: A human-readable operation name.
• Start and End Timestamps: For calculating duration.
• Tags/Attributes: Key-value pairs describing the span (e.g., http.method=GET, db.instance=orders).
• Events: Timed, structured log messages attached to the span.
• Status: Typically OK, ERROR, or UNSET.

Context Propagation is the mechanism that carries the tracing context (the Trace ID, Span ID, and other metadata like sampling decisions) across process and network boundaries. This is the essential glue that connects spans from different services into a single coherent trace. Propagation is typically achieved via headers in HTTP requests, metadata in gRPC calls, or message properties in asynchronous systems (e.g., Kafka, RabbitMQ). Common standardized formats for these headers include:

• W3C Trace Context: A modern, vendor-agnostic standard (traceparent, tracestate headers).
• B3 Propagation: Used by Zipkin (X-B3-TraceId, X-B3-SpanId).
• Jaeger Propagation: Uses headers like uber-trace-id.

Without proper context propagation, each service would create isolated, unrelated traces.

Tags (also called Attributes or Annotations) are key-value pairs attached to a span that provide descriptive metadata about the operation it represents. They are used for filtering, grouping, and querying traces. Tags are typically set at span creation and are not expected to change. Common examples include:

• Semantic Conventions: Standardized keys defined by OpenTelemetry for common operations.
  • http.method: GET, POST
  • http.status_code: 200, 404, 500
  • db.system: postgresql, redis
  • `db.statement**: The sanitized query.
• Business Context: Application-specific data.
  • user.id: 12345
  • order.id: abc-def
  • feature.flag: new_checkout_enabled

Tags turn a generic timing diagram into a queryable, business-relevant dataset.

Span Events (or simply Events) are structured, timestamped log records that are attached to a specific span. They represent meaningful points in time during the span's execution, providing a finer-grained narrative than the span's start and end times. Each event has:

• A name (e.g., cache.miss, exception, message.sent).
• A timestamp.
• Optional attributes (key-value pairs) for additional detail.

Examples:

• An exception event with attributes for the exception type and message.
• A message event in a publish/subscribe flow.
• A retry event indicating a failed attempt and subsequent retry.

Events are crucial for debugging, as they pinpoint the exact moment and context of failures or significant state changes within an operation.

A Span Link connects a span to one or more causally related spans in another trace. This models relationships that are not strictly parent-child. Links are used in asynchronous or batch processing scenarios where a single span is caused by multiple triggering events, or when a span initiates work that will be processed in a separate, distinct trace.

Key use cases:

• Batch Processing: A single batch job span can be linked to the spans of each individual record that was processed, even if those records originated from different user requests (different traces).
• Message Queues: A consumer span processing a message can be linked to the producer span that created the message, which exists in a different trace.
• Fan-out Operations: A span that triggers multiple parallel, independent asynchronous tasks can link to the root spans of those tasks.

A link contains the Trace ID and Span ID (the Span Context) of the linked span. Unlike a parent relationship, a linked span can exist in a completely separate trace and may have even started before the span that links to it.

The fundamental building block of a distributed trace. A span represents a single, named, and timed operation within a workflow (e.g., a database query or an HTTP call). Each span contains:

• Operation Name: A descriptive label.
• Start and End Timestamps: For calculating duration.
• Span Context: A unique trace ID and span ID for linkage.
• Attributes/ Tags: Key-value pairs for dimensional analysis (e.g., http.status_code=200).
• Events: Timed annotations with a message.
• Links: Connections to causally related spans in other traces.

The mechanism by which trace context (trace ID, span ID, sampling flags) is passed across process and network boundaries. This is essential for correlating spans from different services into a single, coherent trace. Propagation is typically implemented via HTTP headers (using standards like W3C Trace-Context) or messaging system metadata. Without proper context propagation, traces become fragmented and lose their end-to-end visibility.

The process of selectively capturing a subset of traces to balance observability detail with system overhead and cost. Common strategies include:

• Head-based Sampling: The sampling decision is made at the start of the trace (e.g., sample 10% of all requests).
• Tail-based Sampling: The decision is made after trace completion based on its characteristics (e.g., sample all traces with errors or high latency).
• Rate Limiting: Sampling a maximum number of traces per second. Effective sampling is critical for managing the volume of telemetry data in high-throughput systems.