Sampling Strategy

The sampling decision point determines when the keep/discard choice is made relative to the request lifecycle.

• Head-based Sampling: The decision is made at the start of a request (e.g., at the first span). A consistent sampling decision (like a random 10%) is propagated via trace context. This is computationally cheap but can discard interesting traces (like slow or erroneous ones) before they are known to be interesting.
• Tail-based Sampling: The decision is made at the end of a request, after all spans are complete. This allows sampling based on the trace's aggregated properties, such as total duration (>5s), error status, or the presence of specific attributes. This is more resource-intensive but preserves critical operational data.

The sampling logic defines the rule or algorithm used to select traces.

• Probabilistic (Random): A fixed percentage of traces are sampled (e.g., 5%). Simple and statistically representative but blind to trace content.
• Rate Limiting: Ensures no more than N traces per second are collected, protecting the backend from traffic spikes.
• Attribute-Based: Samples traces that contain specific key-value pairs in their spans (e.g., http.status_code=500 or service.name=payment-gateway).
• Adaptive/Dynamic: Automatically adjusts the sampling rate based on system load or the volume of high-priority signals (like errors).

Sampling strategies must define their delivery semantics for the telemetry pipeline, impacting data completeness and system reliability.

• At-Least-Once Delivery: The common guarantee for observability data. A sampled trace may be delivered one or more times to the backend. This prevents data loss but requires the backend to handle potential duplicates, often via trace ID deduplication.
• Best-Effort Delivery: Used in high-volume, low-cost scenarios (e.g., UDP-based protocols like StatsD). Data may be dropped under load without retry, favoring performance over completeness.
• Exactly-Once Semantics: A stringent guarantee where each sampled trace is processed precisely once. This is complex to implement and is typically reserved for critical business metrics, not high-volume tracing.

Every sampling strategy introduces a trade-off between observability detail and system resource consumption.

• Agent/Client-Side Overhead: Head-based sampling has minimal overhead on the instrumented application. Tail-based sampling requires buffering spans in memory until the decision is made, increasing memory pressure.
• Collector/Server-Side Load: A central OTel Collector is often used to perform consistent tail-based sampling, offloading decision logic from applications. This collector must be scaled to handle the full, unsampled volume of span data before the sampling filter is applied.
• Storage & Cost Impact: The primary driver for sampling. Reducing trace volume by 90% (a 10% sample rate) directly reduces storage costs and query latency in the observability backend by approximately an order of magnitude.

Sampling logic is deployed within specific components of the telemetry architecture.

• Within SDK/Agent: Simple head-based probabilistic sampling is often configured directly in the OpenTelemetry SDK or vendor agent (e.g., Datadog Agent).
• At the Collector: The OpenTelemetry Collector is the strategic location for sophisticated tail-based sampling using its tail_sampling processor. It provides a unified policy across all services.
• In the Pipeline: Dedicated stream processors like Vector or Fluentd can be configured with sampling rules as data flows through the pipeline, offering flexibility in transformation and routing.

Sampling does not operate in isolation; it interacts with core observability primitives.

• Trace Context Propagation: Essential for head-based sampling. The sampling decision (often a flag) is embedded in the W3C TraceContext headers to ensure all downstream services respect the initial choice.
• Cardinality Management: Sampling works in tandem with attribute pruning to control the explosion of unique time series, which drives costs in metrics systems like Prometheus.
• Checkpointing: For stateful tail-based samplers in stream processors, periodic checkpointing of in-flight trace buffers to durable storage is required for fault tolerance and recovery from failures.

A trace sampling method where the decision to sample (record) an entire request trace is made at the very beginning of the request. This decision, often based on a random factor or a static rule (e.g., 'sample 10% of traces'), is propagated through all subsequent services using the trace context. It is efficient and deterministic but cannot make decisions based on the trace's outcome, such as whether it contained an error or was unusually slow.

• Use Case: High-volume, low-latency services where upfront cost predictability is critical.
• Trade-off: Low overhead but may miss important late-breaking events.

A trace sampling method where the decision to keep or discard a trace is made after the request has fully completed. An aggregator (like the OTel Collector) examines the aggregated properties of all spans in a trace—such as total duration, error status, or specific attributes—before applying a retention rule. This allows for intelligent sampling focused on interesting behavior, like all traces over 1 second or traces that resulted in a 5xx error.

• Use Case: Capturing full context for performance anomalies and errors without storing all data.
• Trade-off: Requires buffering spans in memory until the trace is complete, adding complexity and latency to the pipeline.

A vendor-agnostic proxy for receiving, processing, and exporting telemetry data. It is the central hub for implementing sophisticated sampling strategies. The Collector can perform head-based sampling at its receivers and is essential for tail-based sampling, where it buffers spans, makes retention decisions, and batches data for export. Its pipeline configuration (receivers, processors, exporters) defines the entire data flow and filtering logic.

• Key Function: Executes sampling logic, enriches data, and routes to multiple backends.
• Deployment: Often run as a DaemonSet or sidecar in Kubernetes clusters.

A method of observing requests as they flow through a distributed system. It provides the end-to-end context that sampling strategies act upon. A trace is composed of spans, which represent individual operations. The fidelity of tracing—how many requests are captured and with what detail—is directly controlled by the sampling strategy. Without sampling, the volume of trace data from high-throughput systems would be prohibitive.

• Foundation: Enables performance debugging and dependency analysis across service boundaries.
• Sampling Impact: Determines the statistical representativeness and cost of the tracing dataset.

A critical reliability guarantee in telemetry pipelines where the system ensures an event (like a span or log) is delivered one or more times to its destination. This guarantee is foundational for sampling systems because a sampled event must not be lost due to network or backend failures. Pipelines achieve this through retries and acknowledgments. The trade-off is the potential for duplicate data, which downstream systems must handle idempotently.

• Contrast with Exactly-Once: Less complex and often sufficient for observability data, where occasional duplicates are preferable to data loss.
• Importance for Sampling: Ensures that a decision to keep a valuable trace is not undone by a transport failure.

A fault-tolerance mechanism in stateful stream processors (often used for tail-based sampling). The system periodically records its internal state—such as buffered spans and their metadata—to durable storage. If the processor crashes or restarts, it can recover from the last checkpoint, preventing data loss for in-flight traces being evaluated. This is essential for maintaining the integrity of sampling decisions in the face of infrastructure instability.

• Use in Sampling: Protects buffered trace data in tail-based samplers during node failures or deployments.
• Implementation: Common in pipeline tools like Apache Flink or stateful OTel Collector configurations.