Distributed Tracing

The trace is the overarching record of a single request's end-to-end journey through a distributed system. It is a directed acyclic graph (DAG) of spans, providing a complete, causal view of the transaction.

• Trace ID: A globally unique identifier (e.g., a 128-bit or 256-bit random number) assigned to the entire request, present in all related spans.
• Purpose: Enables performance analysis (latency, bottlenecks) and debugging by reconstructing the full request path across service boundaries.

A span represents a single, named, and timed operation within a trace, corresponding to a unit of work performed by a service (e.g., a function call, database query, or HTTP request).

• Span ID: A unique identifier for the operation within the trace.
• Parent Span ID: Links this span to its parent, establishing causality.
• Timestamps: Contains a start time and a duration.
• Tags/Attributes: Key-value pairs providing contextual metadata about the operation (e.g., http.method=GET, db.query="SELECT...").
• Logs: Timestamped events with additional structured data relevant to the span's lifetime (e.g., exceptions, debug messages).

Context Propagation is the mechanism that carries the trace context (Trace ID, Span ID, sampling decision, and other metadata) across process and network boundaries, enabling the correlation of spans.

• Injection: The tracing system encodes the context into a carrier (e.g., HTTP headers, gRPC metadata, message queues).
• Extraction: The downstream service extracts the context from the carrier and creates child spans linked to the parent.
• Standards: Common protocols include W3C Trace Context (standard for HTTP) and B3 Propagation (originated from Zipkin). This is critical for tracking requests in heterogeneous fleets where services use different languages and frameworks.

The Span Kind describes the role of a span within a trace, indicating whether it represents the initiator of work, an internal processing step, or a server handling a request. This semantic hint aids in visualization and analysis.

• Client: The initiator of an outgoing remote call (e.g., an HTTP client request).
• Server: The receiver of an incoming remote call.
• Producer: The sender of a message to a queue or stream.
• Consumer: The receiver of a message from a queue or stream.
• Internal: A span representing an in-process operation without remote context.

Sampling is the decision-making process that determines which traces are recorded and exported to the observability backend. It is essential for managing cost and performance overhead in high-volume systems.

• Head-based Sampling: The decision is made at the start of the trace (at the root span). Policies include:
  • Always On/Off: Record all or no traces.
  • Probability: Sample a fixed percentage (e.g., 1%) of traces.
  • Rate Limiting: Sample up to N traces per second.
• Tail-based Sampling: The decision is deferred until the trace is complete, allowing sampling based on the trace's outcome (e.g., high latency, errors). This is more complex but captures critical failures.

Instrumentation is the code added to an application to create spans and traces. It can be manual, automatic, or a hybrid.

• Manual Instrumentation: Developers explicitly write code to start and stop spans, add attributes, and log events. This offers maximum control and detail.
• Automatic Instrumentation: Uses agents, SDKs, or frameworks to inject tracing code at runtime, often via bytecode manipulation or middleware hooks. It provides out-of-the-box tracing for common libraries (HTTP clients, databases, RPC frameworks) with minimal code changes.
• In fleet health monitoring, instrumentation captures critical agent vitals (battery SoC, compute load) and diagnostic data as span attributes, linking system health to specific operational traces.

A trace is the complete end-to-end record of a single logical operation. It is composed of a tree of spans, where each span represents a named, timed operation representing a unit of work performed by a single service or agent.

• Trace ID: A globally unique identifier for the entire operation, propagated across all services.
• Span: The fundamental building block, containing:
  • Span ID: A unique identifier for this specific operation.
  • Parent Span ID: Links this span to its caller, forming the trace tree.
  • Name & Timestamps: The operation name, start time, and duration.
  • Tags/Attributes: Key-value pairs for contextual data (e.g., agent_id="AMR-7", location="Aisle-B12").
  • Logs: Timestamped events with structured data relevant to the span.

For a trace to be reconstructed, the trace context must be propagated across all service and network boundaries. This is the mechanism that ties disparate spans together into a coherent trace.

• Carriers: The context is injected into and extracted from carriers, which are protocol-specific parts of inter-service communication.
  • HTTP Headers: (e.g., traceparent, tracestate W3C standards).
  • gRPC Metadata: Key-value pairs in call metadata.
  • Message Queues: Properties in AMQP or Kafka message headers.
• In Fleet Orchestration: Context must flow from the central orchestrator, through task queues, to individual agent controllers, and even down to the robot's firmware modules, creating a complete picture from cloud to edge.

Instrumentation is the code added to an application to create spans for its operations. Auto-instrumentation uses libraries that dynamically inject tracing code without requiring source code changes.

• Manual Instrumentation: Developers explicitly create spans around critical code blocks, offering maximum control and detail.
• Auto-Instrumentation: Libraries for frameworks (e.g., Express.js, Spring Boot, ROS 2) automatically trace common operations like HTTP requests, database calls, or ROS topic subscriptions.
• Fleet-Specific Libraries: In robotics, instrumentation captures:
  • Planning Cycles: Duration of path planning algorithms.
  • Actuation Commands: Time to send motor instructions.
  • Sensor Fusion: Latency in processing LiDAR/IMU data.
  • Network Hop: Communication delay between the edge agent and cloud orchestrator.

Spans are sent to a centralized observability backend for storage, processing, and visualization. This system is responsible for assembling the trace tree and providing query interfaces.

• Key Functions:
  • Collector/Agent: Receives span data (often via OTLP - OpenTelemetry Protocol).
  • Storage: Uses trace databases (e.g., Jaeger, Tempo, Azure Monitor) optimized for high-cardinality time-series data.
  • Query Engine: Allows searching traces by attributes (e.g., error=true, agent_type="forklift").
• Visualization: The Trace View is a Gantt-chart-like diagram showing all spans in a trace, making it easy to identify the critical path and pinpoint which service or agent caused latency or an error.

A common scenario: A 'retrieve item' task is taking too long. A distributed trace reveals the bottleneck.

Example Trace Breakdown:

1. Span 1 (Orchestrator): Receives API request, validates task (5ms).
2. Span 2 (Task Allocator): Runs auction algorithm to assign task to AMR-03 (50ms).
3. Span 3 (Agent Controller for AMR-03):
   • Child Span 3.1: Downloads navigation map (10ms).
   • Child Span 3.2: Path Planning - computes route to bin location (1200ms - BOTTLENECK).
   • Child Span 3.3: Sends movement commands to robot (2ms).
4. Span 4 (AMR-03 Firmware): Executes physical movement (30,000ms).

Insight: The bottleneck is not the physical movement but the path planning. The investigation now focuses on why planning is slow (e.g., congested map, complex obstacle avoidance).

Distributed tracing is most powerful when correlated with metrics and logs, forming the three pillars of observability.

• Tracing + Metrics: A trace showing high latency in a specific microservice can be cross-referenced with that service's CPU utilization or garbage collection metrics from the same time period.
• Tracing + Logs: The span_id can be included in all structured log messages emitted during that span's execution. This allows you to jump from a slow span in the trace viewer directly to the detailed application logs for that specific operation.
• Fleet Health Correlation: A trace of a failed task can be linked to the agent's health score, battery state of charge (SoC), and diagnostic logs from its watchdog timer, providing a holistic view of the failure's root cause.

A continuous, real-time flow of operational data from agents or services to a central collection system. This data includes metrics, logs, and trace spans. In fleet orchestration, telemetry streams from robots and vehicles provide the raw material for monitoring health, performance, and behavior.

• Purpose: Enables real-time dashboards, alerting, and historical analysis.
• Example: A warehouse robot streaming its battery voltage, motor temperatures, and navigation errors every second.

The four fundamental metrics for monitoring the health and performance of any service or system, as defined in Site Reliability Engineering (SRE). They provide a high-level, actionable view.

• Latency: Time taken to service a request.
• Traffic: Demand on the system (e.g., requests per second).
• Errors: Rate of failed requests.
• Saturation: How 'full' a resource is (e.g., CPU, memory, I/O). In distributed tracing, these signals are often derived from aggregated trace data to assess the health of individual services and the entire workflow.

The data processing architecture responsible for collecting, aggregating, transforming, and routing numerical performance data. It works in tandem with distributed tracing to provide a quantitative view of system behavior.

• Components: Include agents, collectors, aggregators, time-series databases (e.g., Prometheus), and visualization tools.
• Relationship to Tracing: While traces show the detailed journey of a single request, metrics provide statistical summaries across all requests, highlighting trends and anomalies.

The process of identifying patterns in system or agent data that deviate significantly from established baselines or expected behavior. Distributed tracing data is a critical input for anomaly detection algorithms.

• Use Case: Detecting a sudden spike in latency for a specific microservice or identifying a robot that is taking abnormally long to complete a navigation segment.
• Techniques: Range from simple statistical thresholding to machine learning models that learn normal patterns from historical trace and metric data.

A structured method of problem-solving used to identify the underlying, fundamental cause of an incident or failure. Distributed tracing is a primary tool for RCA in microservices and distributed fleet systems.

• Process: Traces allow engineers to follow the exact path of a failing request, pinpointing which service introduced an error or latency bottleneck.
• Outcome: Moves beyond symptomatic fixes (e.g., restarting a service) to address the core issue, such as a database query flaw or a misconfigured inter-agent communication protocol.

The practice of writing log messages in a consistent, machine-parsable format (typically JSON) with explicit key-value pairs. This is complementary to distributed tracing.

• Contrast with Tracing: Logs provide detailed, contextual events within a single service, while traces connect those events across service boundaries.
• Integration: Trace IDs are included in structured log entries, allowing logs from all services involved in a request to be correlated and searched together, creating a complete narrative of execution.