Span

Every span is defined by a name (a string describing the operation, e.g., GET /api/user) and a set of unique identifiers. The Span ID uniquely identifies this specific span. Crucially, the span also carries a Trace ID, which is shared by all spans in the same end-to-end request, enabling correlation across services. These IDs are typically 16-byte random values.

A span encapsulates precise timing data, defining its lifecycle within the system.

• Start Timestamp: The nanosecond-precise time the operation began.
• End Timestamp: The time the operation completed.
• Duration: Calculated as (End Timestamp - Start Timestamp). This is the primary metric for performance analysis, revealing latency bottlenecks in database queries, external API calls, or internal functions.

The span context is the immutable trace state that must be propagated across process boundaries. It contains the Trace ID, Span ID, and other metadata like sampling decisions. This context is injected into outbound requests (e.g., via HTTP headers like traceparent) and extracted from inbound requests, which is the mechanism that stitches distributed work into a single trace. The W3C Trace Context standard ensures interoperability between different tracing systems.

The span kind is a semantic attribute that classifies the span's role in a trace, which affects how timing and relationships are interpreted by analysis tools. The primary kinds are:

• SERVER: For the receiver of a remote procedure call (e.g., an HTTP server handling a request).
• CLIENT: For the initiator of a remote procedure call (e.g., an outgoing HTTP client call).
• INTERNAL: For operations within the application boundary where no remote context is propagated.
• PRODUCER/CONSUMER: For messaging systems (e.g., Kafka, RabbitMQ).

Attributes are key-value pairs that describe the span with contextual metadata. They are essential for filtering, grouping, and debugging. Examples include:

• http.method: "GET"
• db.system: "postgresql"
• db.statement: "SELECT * FROM users"
• Custom business data like user.id or order.value.

Events (or Annotations) are timestamped logs within a span's lifetime, such as exceptions (exception.type), debug messages, or significant state changes.

Spans exist within a hierarchy. A parent span represents the broader operation that triggered the current child span. This parent-child relationship forms the trace's call graph.

Span Links connect a span to zero or more causally related spans in other traces, used for modeling batch processing or asynchronous triggers.

Finally, a span status (e.g., OK, ERROR) provides a simple signal of the operation's success or failure, often derived from error attributes.

A span representing an outbound HTTP call from a service to an external API or another microservice. This is a Client span kind.

• Key Attributes: http.method (GET, POST), http.url, http.status_code, peer.service (name of the called service).
• Timing: The span's duration measures the full round-trip time from sending the request to receiving the response.
• Context Propagation: The span's trace context is injected into the HTTP headers (e.g., traceparent) to link to the downstream server span.

A span representing the processing of an incoming HTTP request. This is a Server span kind and is typically the parent of the client span that made the request.

• Key Attributes: http.method, http.route (e.g., /api/v1/users), http.target (the full path), http.status_code.
• Root Spans: Often the root span for a trace if the request originates from an uninstrumented source (like a user's browser or a load balancer).
• Work Context: Encompasses the entire business logic executed to service the request, including any internal function calls or database queries.

A span representing the execution of a single database operation, such as a SELECT, INSERT, or stored procedure call. This is typically an Internal span kind.

• Key Attributes: db.system (postgresql, mysql, redis), db.statement (the query text, often sanitized), db.operation (query, insert), db.name (database name).
• Performance Insight: This span's duration is critical for identifying slow queries and database bottlenecks.
• Nesting: Usually a child span of a server handler or an internal processing span.

Spans representing the publishing to or consumption from a message queue (e.g., Kafka, RabbitMQ, AWS SQS).

• Producer Span (Kind: Producer): Created when a message is sent. Attributes include messaging.system, messaging.destination, and message.id.
• Consumer Span (Kind: Consumer): Created when a message is processed. It is linked to the producer span via a span link, not a parent-child relationship, as they are in separate traces.
• Asynchronous Context: Essential for tracing asynchronous, event-driven workflows where a request triggers a background job.

A span representing a significant unit of work within a service's codebase, such as a core business logic function, algorithm, or external library call. This is an Internal span kind.

• Purpose: Provides granular visibility into the internal call stack of a service, breaking down a server handler's work.
• Attributes: Custom attributes like function.name, business.order_id, or processing.stage.
• Instrumentation: Often requires manual or decorator-based instrumentation to create these fine-grained spans, as they are specific to the application's logic.

A span representing a call to an external service via a Remote Procedure Call (RPC) framework like gRPC or Thrift. This functions similarly to an HTTP client span but for RPC semantics.

• Key Attributes: rpc.system (grpc), rpc.service, rpc.method, rpc.grpc.status_code.
• High Precision: gRPC spans are particularly valuable due to the structured nature of the calls and the prevalence of streaming, which can be broken into multiple spans.
• Propagation: Trace context is propagated through the RPC framework's metadata headers.

A trace is the complete end-to-end record of a single request as it flows through a distributed system. It is composed of a collection of spans that are causally related, forming a directed acyclic graph (DAG). The trace provides the holistic view, while each span represents an individual unit of work within that journey.

• Root Span: The first span in a trace, which has no parent.
• Trace Context: The global state (Trace ID, etc.) that is propagated to keep all spans in a trace correlated.

Span context is the immutable, portable state that must be propagated across process boundaries to link spans together into a trace. It contains the minimal data required to establish causality.

Core components include:

• Trace ID: A globally unique 16-byte identifier for the entire trace.
• Span ID: An 8-byte identifier for the current span.
• Trace Flags: Includes the sampling decision.
• Trace State: Carries vendor-specific tracing information.

This context is injected into carrier formats (like HTTP headers) by a propagator.

Span attributes (or tags) are key-value pairs attached to a span that provide descriptive metadata about the operation it represents. They are essential for filtering, grouping, and debugging.

Common attribute examples:

• http.method: "GET"
• http.url: "https://api.example.com/user"
• db.system: "postgresql"
• `db.statement": "SELECT * FROM users"
• `error.type": "TimeoutError"

Attributes should follow semantic conventions (e.g., from OpenTelemetry) for consistency across instrumentation.

Span kind is a semantic classification that describes the role of a span within a trace. It influences how timing is interpreted and how causality is inferred.

Defined kinds in OpenTelemetry:

• INTERNAL: Represents an operation within an application boundary (default).
• SERVER: The recipient of a remote request.
• CLIENT: The initiator of a remote request.
• PRODUCER: The initiator of a message in a messaging system.
• CONSUMER: The receiver of a message in a messaging system.

For example, a CLIENT span sending an HTTP request should be the parent of a SERVER span receiving it.

Instrumentation is the process of adding code to an application to generate telemetry data like spans. It involves creating spans, adding attributes, recording events, and managing context propagation.

Two primary approaches:

• Manual Instrumentation: Developers explicitly write code to start and end spans using a tracing SDK (e.g., OpenTelemetry API). Offers maximum control.
• Auto-Instrumentation: Uses agents, bytecode manipulation, or wrappers to inject tracing code automatically for common frameworks and libraries (e.g., Express, Django, PostgreSQL drivers). Reduces code changes.

Both methods generate spans that conform to the same tracing model.

W3C Trace Context is a formal W3C recommendation that standardizes how trace context is propagated across HTTP requests and other platforms. It defines specific HTTP headers (traceparent, tracestate) and a consistent value format.

• traceparent: Carries the version, trace-id, parent-id (span-id), and trace flags.
• tracestate: Extends traceparent with vendor-specific data in a multi-tenant format.

This standard ensures interoperability between different tracing systems (e.g., a service instrumented with OpenTelemetry can propagate context to a service using a vendor-specific APM tool).