Span Attributes

Semantic Conventions are standardized, pre-defined attribute keys and values established by the OpenTelemetry project to ensure consistency across instrumentation. They define common attributes for well-known operations, such as:

• http.method: The HTTP request method (e.g., GET, POST).
• db.system: The database management system (e.g., postgresql, redis).
• rpc.service: The name of the RPC service being called. Using these conventions ensures traces from different libraries and services can be aggregated and queried uniformly, which is critical for automated analysis and dashboarding.

Span attributes support a constrained set of primitive value types to ensure efficient serialization, storage, and querying. The allowed types are:

• String: For textual data (e.g., user.id="abc123").
• Integer: For whole numbers (e.g., http.status_code=404).
• Double: For floating-point numbers (e.g., duration.ms=152.7).
• Boolean: For true/false values (e.g., error=true).
• Array of primitives: For lists of uniform values (e.g., http.request.header="["content-type", "authorization"]"). Complex objects must be flattened into key-value pairs or serialized as a string.

Attribute cardinality—the number of unique key-value combinations—directly impacts storage cost and query performance. High-cardinality attributes, like unique user IDs or request IDs, can cause exponential data growth. Best practices include:

• Using low-cardinality attributes for grouping and filtering (e.g., service.name, deployment.environment).
• Being judicious with high-cardinality data; consider logging it separately or using span events.
• Implementing attribute limits in SDKs (e.g., OpenTelemetry's default limit of 128 attributes per span) to prevent accidental data explosions.

Attributes can be added at multiple stages in a trace's lifecycle, known as enrichment points:

1. At Span Creation: Core attributes (e.g., http.method) are set by the auto-instrumentation or manual SDK call.
2. During Processing: Attributes can be added by an OpenTelemetry Collector using processors. For example, the attributes processor can add cloud.region="us-east-1" to all spans.
3. In the Backend: Tracing backends can enrich spans at ingest time using contextual data (e.g., mapping a service.name to a team owner from a CMDB). This layered approach separates operational instrumentation from business context.

Beyond technical metadata, attributes are the primary vector for injecting business context into observability data. This transforms generic traces into actionable, domain-specific insights. Examples include:

• shopping.cart.id: To trace the performance of cart-related operations.
• payment.transaction.type: To segment traces by "refund" vs. "purchase".
• workflow.execution.id: To correlate all spans related to a specific long-running business process. This enables SREs and business analysts to answer questions like "What is the 95th percentile latency for checkout requests from premium users?"

The ultimate value of attributes is realized through queryability. In tracing backends like Jaeger, Tempo, or commercial APMs, attributes become the primary filter dimensions. Effective attribute design enables:

• Precise Filtering: http.status_code=500 AND deployment.environment="prod"
• Aggregation and Grouping: Group trace latency by service.version to validate a new release.
• Statistical Analysis: Calculate error rates for specific customer.tier. Poorly named or inconsistent attributes render traces unsearchable, negating their diagnostic value.

A span is the fundamental unit of work in distributed tracing, representing a named, timed operation for a contiguous segment of work within a service. It is the container to which span attributes are attached.

• Structure: Contains a start/end timestamp, operation name, and status.
• Purpose: Represents operations like a database query, an HTTP handler, or a function call.
• Relationship: Spans are nested to form parent-child relationships, creating a call graph. The attributes on a span provide the descriptive metadata for this unit of work.

A trace is a collection of spans that represents the end-to-end path of a single request as it propagates through a distributed system. It visualizes the causal relationships between spans.

• Graph Structure: Forms a directed acyclic graph (DAG) where spans are nodes.
• Correlation: All spans in a trace share a unique Trace ID.
• Context: The complete trace provides the macro-view; the span attributes on individual spans provide the micro-details necessary to understand each step's context (e.g., http.url for a client call, db.statement for a query).

Span context is the immutable, portable state that must be propagated across process boundaries to link spans together into a trace. It is distinct from the mutable span attributes.

• Core Components: Contains the Trace ID, Span ID, trace flags (e.g., sampling decision), and trace state.
• Propagation: This context is injected into carrier formats (like HTTP headers) and extracted by downstream services.
• Differentiation: While context is for correlation, attributes are for annotation. Context answers "which trace is this?"; attributes answer "what exactly happened in this operation?".

Instrumentation is the process of adding observability code to an application to generate telemetry data like spans, metrics, and logs. It is how span attributes are created and attached.

• Methods: Can be manual (developer-added code) or automatic (via auto-instrumentation agents/SDKs).
• Attribute Creation: Instrumentation libraries automatically add standard attributes (e.g., http.method) and provide APIs for developers to add custom business attributes (e.g., user.id).
• Goal: To create meaningful, well-attributed spans without excessive manual coding overhead.

OpenTelemetry (OTel) is the vendor-neutral, open-source observability framework that provides standardized APIs, SDKs, and tools for generating and managing telemetry, including span attributes.

• Semantic Conventions: OTel defines standardized attribute names and values (e.g., http.status_code) to ensure consistency across different services and tracing backends.
• API: Provides a uniform set_attribute(key, value) method across all supported programming languages.
• Ecosystem: As the de facto standard, using OTel ensures your span attributes are portable across different analysis tools (Jaeger, Zipkin, commercial APMs).

Trace enrichment is the process of adding or modifying contextual metadata on spans after they are generated, often within a processing pipeline like the OpenTelemetry Collector. It works hand-in-hand with initial span attribute instrumentation.

• Use Case: Adding environment-level tags (e.g., deployment.environment=production), sensitive data redaction, or business context derived from other systems.
• Separation of Concerns: Allows developers to instrument core application logic, while platform/SRE teams can add operational context post-hoc.
• Pipeline Stage: A common step in a trace pipeline before data is exported to a backend for storage and analysis.