Tail Sampling

The defining characteristic of tail sampling is that the sampling decision is deferred until after the entire request has completed. This allows the sampling logic to evaluate the complete trace context, including:

• Total request latency
• Final HTTP status code or error state
• All aggregated span attributes from across services
• The presence of specific business logic markers or tags This enables highly informed sampling based on the actual outcome, rather than a probabilistic guess at the start.

Tail sampling uses declarative rules to determine which traces are retained. These rules are evaluated against the complete trace. Common rule types include:

• Latency-based: Keep traces where the total duration exceeds a threshold (e.g., > 2 seconds).
• Error-based: Keep all traces that resulted in an error (HTTP 5xx, application exceptions).
• Attribute-based: Keep traces containing specific span attributes (e.g., customer_tier="premium", http.route="/api/payment").
• Probabilistic: Keep a random percentage of all traces, applied after other rules. Rules are typically executed in a pipeline within a central collector like the OpenTelemetry Collector.

Tail sampling is almost always implemented in a centralized telemetry processor, not within individual services. The OpenTelemetry Collector is the canonical implementation, using its tail_sampling processor. The workflow is:

1. All services emit 100% of traces to the collector.
2. The collector buffers traces for a configurable period.
3. Once a trace is considered "complete," the collector evaluates it against the defined sampling policy.
4. Traces matching the policy are exported to the backend (e.g., Jaeger, Datadog); others are discarded. This architecture prevents sampling bias and ensures consistent rule application across the entire system.

This strategy is engineered for debugging and Service Level Objective (SLO) monitoring, not for reducing upstream data volume. Its core value is in guaranteeing the retention of diagnostically valuable traces that would be randomly missed by head sampling.

• Debugging: Ensures all error traces and high-latency outliers are captured for root cause analysis.
• SLO Monitoring: Enables reliable calculation of error rates and latency percentiles (p95, p99) from the sampled data, as the sample is not random but criteria-based.
• Cost Efficiency: While it processes 100% of traces initially, it dramatically reduces long-term storage costs by discarding uninteresting, fast-successful traces.

Tail sampling introduces specific engineering trade-offs:

• Decision Latency: Traces are not available in the backend in real-time. They are delayed by the collector's decision_wait time (e.g., 10-30 seconds) as it waits for slow spans to arrive.
• Collector Resource Load: The collector must buffer and evaluate every single trace, requiring significant memory and CPU resources, especially under high request volumes.
• Trace Completeness Risk: If the collector crashes or restarts, all buffered traces that haven't been evaluated are lost. This requires careful deployment with high availability and persistent buffers.

Effective tail sampling combines multiple rules into a policy. A standard production policy might be:

• Always keep error traces (status_code == ERROR).
• Keep slow traces (latency > 1s).
• Keep a tiny percentage of all successful traces (e.g., 0.1% probabilistic) for general traffic shape monitoring.
• Always keep traces for critical user journeys (e.g., where span.attributes.user_id is in a premium list). This layered approach ensures comprehensive coverage for incidents while maintaining a manageable data volume. The policy is typically defined in the collector's configuration YAML.

Organizations use tail sampling to guarantee visibility into key user journeys or high-value transactions. Rules are based on business-level attributes added to spans via enrichment.

• Example: Sample 100% of traces where transaction.type equals checkout and cart.value exceeds $1000.
• Benefit: Provides deterministic observability for critical revenue-generating or compliance-sensitive workflows, ensuring they are never missed due to probabilistic sampling.

In regulated industries, tail sampling acts as an audit trail for security-sensitive operations. It ensures a complete record of requests accessing protected resources or exhibiting suspicious patterns.

• Example: Retain all traces where user.role equals admin and access.target contains /financial/records.
• Benefit: Creates an immutable, end-to-end forensic log of privileged access or potential security events, which is essential for post-incident analysis and regulatory compliance reporting.

During progressive rollouts, tail sampling is configured to capture a higher percentage—or all—traces from the new deployment variant. This is often combined with trace-by-trace comparisons.

• Example: Sample 50% of all traces from the stable service, but 100% of traces from the canary service tagged with deployment=canary-v2.
• Benefit: Provides statistically significant, high-fidelity data to compare latency, error rates, and behavior between versions, enabling confident release decisions.

A trace is the complete end-to-end record of a single request as it flows through a distributed system. It is composed of a collection of spans that form a directed acyclic graph (DAG), showing the causal relationships between operations. Tail sampling makes its keep/discard decision based on the full context of a completed trace, such as its overall latency or error status.

A span is the fundamental building block of a trace, representing a single, named, and timed operation within a service (e.g., a database query or an HTTP call). Each span contains:

• Span Attributes: Key-value metadata (e.g., http.status_code=500).
• Span Kind: Semantic role (Client, Server, Internal).
• Timing data: Start and end timestamps. Tail sampling evaluates the aggregate of all spans in a trace to make its sampling decision.

Trace sampling is the overarching practice of selectively capturing a subset of traces to manage data volume, storage costs, and processing overhead. Tail sampling is one specific strategy within this practice. Other primary strategies include:

• Head Sampling: The sampling decision is made at the start of a request (e.g., simple random sampling).
• Tail Sampling: The decision is made at the end of a request, based on its full attributes. The choice between head and tail sampling is a core trade-off between upfront efficiency and retrospective, criteria-based capture.

The OpenTelemetry Collector is a vendor-agnostic proxy that receives, processes, and exports telemetry data. It is the most common architectural component where tail sampling logic is implemented. The collector runs processors that can:

• Ingest traces via OTLP.
• Buffer traces until they are complete.
• Apply sampling rules based on span attributes (e.g., error=true or duration > 2s).
• Forward only the sampled traces to backends like Jaeger or commercial APM tools.

Trace enrichment is the process of adding contextual metadata to spans after they are generated. This often occurs in the same pipeline stage as tail sampling. Enrichment can add business context (e.g., user tier, transaction value) or deployment metadata (e.g., pod name, version) that can then be used as criteria in tail sampling rules. For example, a rule could be configured to sample 100% of traces where user.tier=premium and http.status_code=500.

Distributed context propagation is the mechanism that carries the trace context (containing the Trace ID and Span ID) across service boundaries via HTTP headers, gRPC metadata, or message queues. This mechanism is essential for tail sampling because it ensures all spans from a single request are correlated into one trace. Standards like W3C Trace Context ensure interoperability, allowing the tail sampling processor to see the unified, end-to-end request.