Jaeger

To manage the immense volume of trace data in production, Jaeger clients and collectors implement sophisticated trace sampling. This includes:

• Probabilistic (Head) Sampling: A fixed percentage of traces are sampled at the request's start.
• Rate-Limiting Sampling: Controls the number of traces per second.
• Remote-Sampled Configuration: Sampling strategies can be dynamically configured and fetched from a central Jaeger backend, allowing policies to be updated without redeploying services.

This ensures high-value traces (e.g., those with errors or high latency) are retained while controlling storage costs.

Jaeger's architecture separates the query service from pluggable storage layers, enabling it to scale with different database technologies optimized for trace data.

• Cassandra: The default, battle-tested backend for scalable, write-heavy workloads.
• Elasticsearch: Popular for its powerful full-text search and aggregation capabilities on trace attributes.
• gRPC Plugin: Supports custom storage implementations via a well-defined gRPC interface.

This design allows deployment flexibility, from self-managed Cassandra clusters to managed Elasticsearch services in the cloud.

Jaeger automatically generates service graphs by analyzing trace data. These visual maps show:

• Service Nodes: All services involved in request flows.
• Directed Edges: The calls between services, annotated with metrics like request rates and error percentages.
• Dynamic Updates: Graphs update near-real-time as new trace data is ingested.

This feature is critical for understanding system topology, identifying unexpected dependencies, and visualizing the impact of failures across a microservice mesh.

A powerful feature for performance debugging, Jaeger's UI allows for side-by-side comparison of two traces. Engineers can:

• Select Traces: Choose two traces by Trace ID.
• Visual Diff: View a unified timeline highlighting differences in span durations and structure.
• Root-Cause Investigation: Quickly pinpoint which service or operation caused a regression in latency between two executions of the same logical request, such as before and after a deployment.

Jaeger is designed for modern infrastructure, offering multiple deployment patterns to suit different operational scales.

• All-in-One: A single binary with UI, collector, query, and embedded storage for local development and testing.
• Production Modular: Components (Agent, Collector, Query, UI) are deployed as separate, scalable services.
• Kubernetes-Native: Official Helm charts and operator (Jaeger Operator) manage the lifecycle, configuration, and provisioning of Jaeger instances on Kubernetes, supporting features like sidecar agent injection.

This flexibility supports evolution from a simple pilot to a large-scale, enterprise-grade tracing platform.

A span is the fundamental building block of a trace in Jaeger. It represents a single, named, and timed operation representing a contiguous segment of work within a service. Spans are nested and ordered to model the flow of a request. Key characteristics:

• Timing: Contains a start timestamp and a duration.
• Context: Holds a span ID and a trace ID for correlation.
• Metadata: Includes span kind (e.g., client, server), attributes (key-value pairs like http.method=GET), status codes, and optional events or logs.
• Relationships: Spans have parent-child relationships, forming a hierarchy. A root span initiates a trace, and child spans represent downstream operations.

A trace is a directed acyclic graph (DAG) of spans that represents the end-to-end journey of a single request as it traverses a distributed system. In Jaeger, a trace is the complete story assembled from all correlated spans. Essential aspects:

• Global Identifier: Unified by a globally unique Trace ID propagated across all services.
• Visualization: Jaeger's UI visualizes traces as timeline views or flame graphs, where the width of a bar represents duration, showing critical paths and bottlenecks.
• Purpose: Traces answer questions about request latency, error propagation, and service dependencies, moving debugging from single-service to system-wide analysis.

This is the mechanism that enables tracing across service boundaries. For Jaeger to construct a complete trace, the trace context (Trace ID, Span ID, sampling flags) must be propagated from one service to the next. Key mechanisms:

• Inject/Extract: The tracing library injects context into outbound requests (e.g., as HTTP headers) and extracts it from inbound requests.
• Standards: Common formats include W3C Trace Context (modern standard) and B3 Propagation (originating from Zipkin). Jaeger clients support multiple propagators.
• Critical Function: Without proper propagation, spans appear as disconnected, orphaned operations, breaking the end-to-end view.

A service graph is a topological map automatically derived from trace data that shows the services in a system and the directional request flows (dependencies) between them. Jaeger can generate service graphs to provide a high-level architectural view. Key features:

• Dynamic Discovery: The graph is not manually configured; it is inferred by observing actual traffic patterns captured in traces.
• Operational Insights: Reveals unexpected dependencies, highlights services with high error rates or latency, and aids in impact analysis during incidents.
• Derived Metric: Often includes edge metrics like request counts per minute and error rates between services, turning trace data into system topology intelligence.