Span

The span name is a human-readable identifier for the operation (e.g., HTTP GET /api/users). The span kind categorizes the role of the span within a trace:

• SERVER: For the receiver of a remote request.
• CLIENT: For the initiator of a remote request.
• INTERNAL: For operations within an application boundary.
• PRODUCER & CONSUMER: For messaging systems. The name and kind are essential for semantic grouping and understanding the flow of a request.

The trace context is the causal metadata that links spans together. It contains two critical identifiers:

• Trace ID: A globally unique 16-byte identifier shared by all spans in a single distributed trace.
• Span ID: An 8-byte identifier unique within its trace for the specific span. This context is propagated (e.g., via HTTP headers) across service boundaries, enabling the reconstruction of the full request path. A parent span ID explicitly defines the causal relationship within the trace tree.

Spans are fundamentally temporal units. They record precise timestamps:

• Start Timestamp: The nanosecond-precision UTC time when the operation began.
• End Timestamp: The time when the operation completed. The duration is calculated as (End Timestamp - Start Timestamp). This allows for precise latency analysis of individual operations and aggregate performance of entire traces. High-resolution timestamps are critical for identifying performance regressions.

The span status indicates the final outcome of the operation:

• Unset: The default, for successful operations.
• Ok: Explicitly set for successful operations.
• Error: Indicates the operation failed. Must be set for errors. Span events (or annotations) are timestamped logs attached to a span, representing significant moments during its execution (e.g., exception thrown, cache.miss, message.sent). They provide a detailed, time-ordered narrative of the span's internal lifecycle.

Span attributes are key-value pairs that describe the context of the operation. They are the primary mechanism for adding dimensional metadata for filtering and aggregation. Examples include:

• http.method: "GET"
• db.system: "postgresql"
• net.peer.ip: "10.0.0.1"
• custom.business.id: "order_12345" Attributes should follow semantic conventions for consistency but can be extended with custom, business-specific data. They are indexed in observability backends for powerful querying.

A span link creates a causal relationship between a span and another span in a different trace. This is crucial for modeling batch or asynchronous processing where a single operation is triggered by multiple initiating requests. Example: A background job that processes messages from a queue can create a span linked to the spans from each producer that enqueued those messages. Each link contains the Trace ID and Span ID of the linked context. Links differ from parent-child relationships, which exist within a single trace.

Distributed tracing is the overarching methodology for observing and profiling requests as they flow through a distributed system. It tracks the full path, latency, and relationships between operations across multiple services and components.

• A trace is the complete end-to-end record of a single request.
• Spans are the individual, timed operations that compose a trace.
• This technique is critical for diagnosing latency issues and understanding service dependencies in microservices and agentic architectures.

OpenTelemetry is a vendor-neutral, open-source observability framework that provides the standardized tools to generate, collect, and export telemetry data, including spans. It is the de facto standard for instrumenting modern applications.

• Provides unified APIs and SDKs for creating spans and traces.
• Defines the OpenTelemetry Protocol (OTLP) for data transmission.
• Includes the OTel Collector for receiving, processing, and routing telemetry data.

Trace context is the metadata that propagates the necessary identifiers to correlate spans from different services into a single, coherent distributed trace. It is essential for maintaining the integrity of a request's journey.

• Contains the Trace ID (unique to the overall request) and Span ID (unique to the current operation).
• Often propagated via HTTP headers (following the W3C TraceContext standard) or RPC metadata.
• Ensures that all spans generated by downstream services are linked to the correct parent trace.

Span attributes (or tags) are key-value pairs that provide descriptive, queryable metadata about a specific operation. They turn raw timing data into actionable, contextualized information.

• Examples: http.method="GET", db.system="postgresql", agent.decision="retry", error=true.
• Used for filtering, grouping, and analyzing trace data in observability backends.
• Critical for adding business context (e.g., user.id, transaction.amount) to operational telemetry.

A span event (or annotation) is a structured log message attached to a specific point in time within a span's lifetime. It records notable occurrences during the operation's execution.

• Examples: Recording an exception stack trace, a checkpoint in a long-running process, or a state change within an agent's reasoning loop.
• Contains a timestamp and a set of attributes describing the event.
• Provides a finer-grained, time-series view of what happened during the span, not just at its start and end.

A sampling strategy is a rule-based approach for selectively reducing the volume of trace data collected, balancing observability detail against system overhead and cost.

• Head-based sampling: The decision to sample is made at the start of the trace (e.g., sample 10% of all requests). Simple but can miss important, rare traces.
• Tail-based sampling: The decision is made after the trace completes, based on its properties (e.g., keep all traces with errors or latency > 1s). More intelligent but requires buffering.
• Essential for managing data volume in high-throughput agentic systems.