Head Sampling

The sampling decision is made at the very beginning of a request, typically by the root service or an edge proxy/load balancer. This decision is based on initial request metadata (e.g., a random number, a specific header value, or the user ID) before any significant work is performed. The chosen sampling rate (e.g., 10%) is applied uniformly at this ingress point.

• Key Benefit: Eliminates the overhead of making sampling decisions in every downstream service.
• Key Limitation: Cannot sample based on the request's outcome, such as high latency or an error, as those are unknown at the start.

Once the sampling decision is made at the request's head, it is immutable and propagated throughout the entire request path. All services involved in processing that request respect the initial decision, ensuring the trace is either fully captured or fully dropped. This is achieved by setting the trace flag within the W3C Trace Context headers.

• A sampled=true flag instructs all instrumented services to record their spans.
• A sampled=false flag tells services to bypass span creation for that request, minimizing instrumentation overhead.

Head sampling provides a predictable, fixed cost for tracing overhead. The computational cost is directly proportional to the configured sampling rate. For example, a 5% sampling rate means approximately 5% of requests incur the full cost of span creation, serialization, and export.

• Low & Stable Overhead: Ideal for high-throughput systems where consistent performance is critical.
• No Post-Processing Delay: Unlike tail sampling, there is no need to buffer traces for later evaluation, reducing memory pressure on collectors.
• Trade-off: This fixed cost is incurred regardless of whether the sampled traces are interesting or useful for analysis.

Implementation relies on distributed context propagation. The root service uses a propagator to inject the trace context, including the sampling decision, into outbound requests (e.g., via HTTP headers).

Common Implementation Points:

• API Gateways / Load Balancers: (e.g., NGINX, Envoy with OpenTelemetry modules).
• Application Frameworks: Initial middleware in a web server (e.g., Express.js, Spring Boot).

The propagated W3C traceparent header includes the trace-flags field, where the least significant bit represents the sampling decision (01 = sampled, 00 = not sampled).

Head sampling is the default and most common strategy for general-purpose tracing in production.

Ideal for:

• High-Volume Health Monitoring: Getting a continuous, statistically representative sample of system behavior.
• Latency Analysis: Understanding the distribution of request performance across services.
• Dependency Mapping: Building accurate service graphs by observing a steady flow of sampled requests.
• Environments with Strict Performance Budgets: Where the overhead of more adaptive sampling is prohibitive.

Head Sampling and Tail Sampling represent two fundamental philosophical approaches to trace capture.

Head sampling is about volume control; tail sampling is about content filtering.

Tail sampling is a trace sampling strategy where the decision to retain or discard a trace is deferred until after the request has completed. A processor (often the OpenTelemetry Collector) evaluates the full set of span attributes—such as duration, error status, or custom tags—against a set of rules before making the final sampling decision.

• Key Mechanism: Decisions are based on the complete outcome of the request.
• Primary Use Case: Capturing all traces that meet specific criteria of interest (e.g., all errors, all requests over 1s latency) regardless of initial sampling probability.
• Trade-off: Requires buffering trace data until the decision can be made, increasing memory and processing overhead compared to head sampling.

Trace sampling is the overarching process of selectively capturing a subset of request traces to balance observability detail with data volume, storage cost, and processing overhead. It is a critical function in production systems where capturing 100% of traces is often prohibitively expensive.

• Core Objective: Manage the trade-off between insight and cost.
• Common Strategies: Includes head-based (decision at start), tail-based (decision at end), and rate-based (fixed percentage) sampling.
• Sampling Rate: The percentage of requests that are traced (e.g., 1%, 10%). This is often adjusted dynamically based on system load.

Distributed context propagation is the mechanism that carries the trace context—including the Trace ID, Span ID, and sampling decision—across service boundaries. This is what makes distributed tracing possible, ensuring all spans from a single request can be correlated.

• Propagation Formats: Standards like W3C Trace Context (modern standard) and B3 Propagation (Zipkin format) define the header structures.
• The Propagator: A library component that injects context into outbound requests (e.g., HTTP headers, gRPC metadata) and extracts it from inbound requests.
• Critical for Head Sampling: The initial sampling decision made by the root service must be propagated to all downstream services, instructing them to respect that decision.

A trace is the complete end-to-end record of a single request's journey through a distributed system. It is composed of a collection of spans that form a directed acyclic graph (DAG), showing the causal and temporal relationships between operations.

• Trace ID: A globally unique identifier that ties all spans of a single request together.
• Representation: Visualized as a timeline or a flame graph, where the width of bars represents duration.
• Sampling Unit: Head and tail sampling strategies make a binary decision at the trace level—to sample the entire trace or discard it entirely.

A span represents a single, named, and timed operation within a trace, such as a function call, database query, or HTTP request to another service. Spans are the fundamental building blocks of a trace.

• Span ID: A unique identifier for the operation within its trace.
• Parent-Child Relationships: Spans can be nested, representing call stacks and asynchronous operations.
• Attributes: Key-value pairs (e.g., http.method=GET, db.statement) that provide metadata about the operation.
• In Head Sampling: Once a trace is sampled, all subsequent spans for that request are collected and emitted.