Span Kind

A CLIENT span represents the initiator of an outbound synchronous request to a remote service. It models the work done while the client is waiting for a response.

• Timing Interpretation: The span's duration measures the outbound network latency plus the remote service's processing time.
• Typical Operations: HTTP client calls, gRPC requests, database queries initiated by the application.
• Relationship: A CLIENT span is typically the parent of a corresponding SERVER span in the downstream service.

A SERVER span represents the work done by a service to handle an incoming synchronous request. It is the counterpart to a CLIENT span.

• Timing Interpretation: The span's duration measures the internal processing time of the request within this service, excluding network latency from the client.
• Typical Operations: Handling an incoming HTTP request, processing a gRPC call, executing a serverless function trigger.
• Relationship: A SERVER span is typically the child of a CLIENT span from the upstream caller.

A PRODUCER span represents the creation and submission of a message to a messaging system (queue, topic, stream) in an asynchronous, fire-and-forget operation.

• Timing Interpretation: The span's duration measures the time to construct and enqueue the message. It does not include the time for a consumer to process it.
• Typical Operations: Publishing to Apache Kafka, sending to RabbitMQ, putting a job on a Redis queue.
• Relationship: A PRODUCER span is linked (via Span Links) to one or more CONSUMER spans that later process the message, which may be in entirely different traces.

A CONSUMER span represents the reception and processing of a message from a messaging system. It is the asynchronous counterpart to a PRODUCER span.

• Timing Interpretation: The span's duration measures the processing time for the message after it is dequeued. The start time is when processing begins, which may be long after the message was produced.
• Typical Operations: Consuming from a Kafka topic, processing an SQS message, handling a background job.
• Relationship: A CONSUMER span is linked (via Span Links) back to the PRODUCER span that created the message, creating a causal relationship across asynchronous boundaries.

An INTERNAL span represents an operation contained entirely within the bounds of a single service or application, with no remote communication. This is the default span kind in many tracing systems.

• Timing Interpretation: The span's duration measures pure computation time—the execution of business logic, function calls, or internal data processing.
• Typical Operations: Business logic functions, algorithm execution, database transactions (when the DB driver is instrumented as part of the app), cache lookups.
• Relationship: INTERNAL spans form the nested call stack within a service, with parent-child relationships defined by the local execution flow.

Misclassifying a database call or internal RPC can drastically skew performance analysis. The key difference is boundary crossing.

• Use CLIENT when the operation involves:

  • A network call to another separately deployed service (microservice, external API).
  • A call to a database or cache that is instrumented as a separate entity (e.g., using a database-specific tracing client). The span duration includes network round-trip.

• Use INTERNAL when the operation is:

  • A local function or method call.
  • An in-process operation where the 'remote' component (like an embedded database) is considered part of the same service boundary for tracing purposes.

Correct classification ensures latency attribution is accurate: network time is isolated to CLIENT spans, while processing time is isolated to INTERNAL and SERVER spans.

A Span Kind is a required semantic attribute defined by the OpenTelemetry specification. It describes the relationship of the span to its remote parent or child. The primary standardized kinds are:

• CLIENT: Describes a span that initiates a remote operation (e.g., an agent making an outbound HTTP API call).
• SERVER: Describes a span that handles a remote operation initiated by a client.
• PRODUCER: A span that initiates an asynchronous operation, like sending a message to a queue.
• CONSUMER: A span that processes the result of an asynchronous operation.
• INTERNAL: The default. Describes an operation within the service boundary with no remote parent or child. This classification allows observability backends to correctly build service graphs and calculate network latency.

Span Kind directly informs how the observability system calculates critical latency metrics. For a CLIENT span, the duration includes the full round-trip time of the network call. For a SERVER span, the duration measures the time spent processing the request internally. In agentic systems, this distinction is vital. If an agent's tool-calling span is mislabeled, latency attribution becomes incorrect. For example, a CLIENT span duration for a database call includes network time, while an INTERNAL span for a local vector search within the agent's memory only measures compute time. This precision is required for accurate performance benchmarking and SLO definition.

Observability platforms use Span Kind to automatically infer service topology and dependencies. A CLIENT span from Service A linked to a SERVER span in Service B creates a directed edge in the service graph. For multi-agent systems, this enables visualization of the agent interaction graph. You can see:

• An Orchestrator agent (CLIENT) calling a Specialist agent (SERVER).
• An agent (PRODUCER) emitting an event that triggers a downstream workflow (CONSUMER). Without correct Span Kind, these critical architectural relationships and potential bottlenecks remain hidden, hindering multi-agent observability.

Correctly instrumenting an agent requires deliberate Span Kind assignment for different operations:

• Planning/Reasoning Loops: Typically INTERNAL spans, as they are internal cognitive processes.
• External Tool/API Execution: Must be CLIENT spans. The external service's handler should be a SERVER span.
• Inter-Agent Communication: When Agent A calls Agent B via RPC/HTTP, A's span is CLIENT, B's is SERVER. For asynchronous message passing (e.g., via a queue), use PRODUCER/CONSUMER.
• Database/Vector Store Queries: CLIENT spans, as they are calls to an external service. Mislabeling an external call as INTERNAL will hide its network latency and break dependency mapping.

Analyzing spans by their kind is a primary debugging technique. In a trace showing high latency:

1. Filter for CLIENT spans to identify slow external dependencies (APIs, databases, other agents).
2. Examine long SERVER spans to find internal processing bottlenecks within a specific agent (e.g., a complex reasoning step).
3. Look for PRODUCER spans without linked CONSUMER spans to detect lost messages or broken asynchronous workflows in agent choreography. This structured analysis, powered by Span Kind, transforms a flat list of spans into a diagnosable map of an agent's end-to-end execution path.

Span Kind works in concert with Span Context propagation. The kind influences how context is propagated:

• A CLIENT span must inject its context (e.g., via HTTP headers) so the downstream SERVER can extract it and continue the trace.
• An INTERNAL span does not propagate context externally.
• PRODUCER spans inject context into message metadata (e.g., Kafka headers) for the CONSUMER to extract. In agentic systems, ensuring W3C Trace Context headers are correctly injected on all CLIENT calls is essential for maintaining trace continuity across agent boundaries and external tools.

A Span is the fundamental unit of work in distributed tracing. It represents a named, timed operation corresponding to a contiguous segment of work within a single service.

• Core Building Block: Every trace is composed of one or more spans.
• Represents Operations: Can be a function call, database query, HTTP request, or any logical unit of work.
• Contains Metadata: Includes a start/end timestamp, Span Kind, status, attributes, and links to other spans.
• Hierarchical Structure: Spans have parent-child relationships, forming a call tree within a trace.

A Trace is a collection of spans that represents the complete end-to-end path of a single request as it propagates through a distributed system.

• Request Journey: Records the lifecycle of a transaction across service and process boundaries.
• Directed Acyclic Graph (DAG): Spans are linked by parent-child relationships, forming a graph of the request flow.
• Correlated by Trace ID: All spans in a trace share a globally unique Trace ID.
• Purpose: Enables performance analysis (latency bottlenecks) and root cause diagnosis of failures across microservices.

Span Context is the immutable tracing state that must be propagated across process boundaries to link spans into a coherent trace.

• Critical for Propagation: Contains the essential identifiers needed for distributed tracing.
• Core Components:
  • Trace ID: The global identifier for the trace.
  • Span ID: The identifier for the current span.
  • Trace Flags: Includes the sampling decision.
  • Trace State: Carries vendor-specific tracing information.
• Carrier Formats: This context is serialized into headers (e.g., W3C TraceContext, B3) for HTTP/gRPC/RPC calls.

Distributed Context Propagation is the mechanism by which trace context (Trace ID, Span ID) is passed between services to maintain continuity across a distributed transaction.

• Maintains Trace Continuity: Allows a trace to be followed from service A to service B to service C.
• Implementation via Propagators: Libraries use Propagator components to inject context into outbound requests and extract it from inbound requests.
• Standard Formats: Uses standardized header formats like W3C Trace Context or B3 Propagation.
• Essential for End-to-End Tracing: Without proper propagation, traces break at service boundaries, creating disjointed segments.

Span Attributes are key-value pairs attached to a span that provide descriptive, queryable metadata about the operation it represents.

• Enriches Observability: Adds context beyond timing and relationships.
• Common Examples:
  • http.method: "GET"
  • http.url: "https://api.example.com/users"
  • db.system: "postgresql"
  • `db.statement": "SELECT * FROM orders"
  • Custom business attributes: user.id, shopping.cart.id.
• Different from Span Kind: While Span Kind is a specific, enumerated semantic hint, attributes are general-purpose metadata.