Observability

Logs are immutable, timestamped records of discrete events that occurred within a robotic system. They provide a detailed, sequential audit trail for post-hoc forensic analysis.

• Structured vs. Unstructured: Modern systems use structured logs (e.g., JSON) with consistent key-value pairs for automated parsing, as opposed to unstructured text.
• Use Case: Debugging a robotic arm's failure to grasp an object by reviewing logs from the perception node (object detection confidence), planning node (calculated grasp pose), and control node (joint torque readings).
• Critical Property: Must include sufficient context (e.g., robot ID, task ID, sensor timestamp) to correlate events across distributed subsystems.

Metrics are numerical measurements collected over intervals of time, representing the quantitative state of a system. They are optimized for real-time alerting and trend analysis.

• Types: Include counters (monotonically increasing values, like total tasks executed), gauges (instantaneous measurements, like CPU temperature), and histograms (statistical distributions, like control loop latency).
• Robotics Examples:
  • Cycle Time: Latency of the primary perception-planning-actuation loop.
  • Power Consumption: Current draw from the battery system.
  • Localization Error: Drift between estimated and ground-truth position.
• Visualization: Typically displayed on dashboards (e.g., Grafana) as time-series graphs for operational monitoring.

Traces record the end-to-end journey of a single operation or transaction as it propagates through a distributed system. They are essential for understanding causality and identifying performance bottlenecks.

• Span: The basic unit of a trace, representing a named, timed operation of work (e.g., "LiDAR point cloud processing"). A trace is a directed acyclic graph of spans.
• Context Propagation: A unique trace ID is passed between all software components (e.g., from the navigation stack to the motor controller) to link related activities.
• Robotics Application: Tracing a "Navigate to Goal" command from high-level mission planner, through global/local planners, to wheel motor commands, capturing latency at each stage and identifying if slowdowns are in perception, planning, or control.

Observability transforms raw telemetry into actionable insights. Consider a warehouse robot fleet experiencing intermittent navigation failures.

1. Alert: A metric for "localization resets per hour" exceeds a threshold.
2. Investigation: Engineers query logs from the failing robots, filtering for ERROR severity around the alert time, finding entries about "AMCL particle cloud collapse."
3. Root Cause Analysis: They examine traces of specific failed navigation requests, discovering that spans for "laser scan matching" are taking >500ms, coinciding with periods of high warehouse traffic causing sensor occlusion.
4. Resolution: The issue is identified as a sensor fusion problem under low-feature environments, leading to a software update that weights the inertial measurement unit (IMU) data more heavily during these conditions.

Telemetry is the broader practice of collecting and transmitting data from remote or inaccessible sources (like a robot in the field) to an IT system for monitoring. Observability is the goal; telemetry (logs, metrics, traces) is the data that makes it possible.

• In Robotics: Encompasses both the three pillars and raw sensor data streams (e.g., camera images, LiDAR point clouds, joint encoder values) that may be sampled for diagnostic purposes.
• Challenge: The volume of raw sensor telemetry is enormous. Effective observability requires intelligent sampling, aggregation, and derivation of higher-level metrics from this raw data.

Implementing observability requires a cohesive toolchain.

• Collection & Export: OpenTelemetry is the dominant open-source standard for generating, collecting, and exporting telemetry data. It provides vendor-agnostic APIs and SDKs for logs, metrics, and traces.
• Transport & Storage: Tools like Prometheus (for metrics), Loki (for logs), and Jaeger or Tempo (for traces) handle aggregation, storage, and querying.
• Visualization & Alerting: Grafana is the prevalent platform for building dashboards that unify data from all sources and configuring alert rules.
• Robotics Middleware: ROS 2 has built-in tools like ros2 tracing (using LTTng) for collecting trace data from the DDS communication layer.

Observability is built on three core data types, each providing a different lens into system behavior.

• Logs: Time-stamped, immutable records of discrete events (e.g., INFO: Motor driver initialized, ERROR: CAN bus timeout). They provide a narrative of what happened.
• Metrics: Numeric measurements aggregated over time (e.g., CPU temperature, wheel odometry error, control loop frequency). They quantify system health and performance.
• Traces: Records of the end-to-end journey of a single operation as it propagates through distributed components (e.g., tracing a pick-and-place command from perception through planning to actuation). They reveal latency bottlenecks and dependency failures.

In robotics, these pillars must be synchronized with a high-precision clock (using protocols like PTP) to correlate physical events across hardware and software boundaries.

For a robot, a core 'internal state' is its pose (position and orientation). Observability here means instrumenting the state estimation pipeline.

• Raw Sensor Streams: Telemetry from IMUs, LiDAR, cameras, and wheel encoders must be logged with precise timestamps.
• Filter Outputs: Key outputs from algorithms like the Kalman Filter or particle filter—estimated pose, covariance matrices (uncertainty), and innovation residuals—are critical metrics. A growing covariance indicates declining observability of the robot's true state.
• Ground Truth Comparison: In testing, metrics like Absolute Trajectory Error (ATE) and Relative Pose Error (RPE) are calculated by comparing the estimated state against motion capture or GPS ground truth, quantifying the filter's performance.

Deterministic execution is non-negotiable. Observability provides verification by instrumenting the perception-planning-action cycle.

• Loop Timing Metrics: Jitter (variance in period) and latency (end-to-end computation time) for each cycle must be measured. Violations of Worst-Case Execution Time (WCET) assumptions can cause system failure.
• Actuator Command & Feedback: Log desired vs. actual joint positions, velocities, and torques. Discrepancies indicate tracking errors, mechanical slippage, or controller instability.
• Queue Depths & Buffer Overflows: Monitor the fill levels of data queues between processes (e.g., image processing to planner). Overflows signal that a component cannot keep up with real-time demands, breaking the system's temporal assumptions.

A primary use of observability is quantifying and bridging the reality gap. When a policy trained in simulation fails on physical hardware, observability data pinpoints why.

• Differential Telemetry: Run identical tests in physics-based simulation (e.g., NVIDIA Isaac Sim) and on the real robot. Systematically compare the telemetry streams: actuator commands, sensor readings, and internal state estimates.
• Identifying Discrepancies: Common gaps appear as unmodeled sensor noise, latency in communication buses, or inaccurate physics parameters (e.g., friction coefficients). Observability metrics make these gaps measurable, guiding iterative improvements to the simulation model or the use of domain randomization.
• This process is foundational for reliable Sim-to-Real transfer.

Observability enables autonomous Fault Detection and Diagnostics (FDD).

• Anomaly Detection Baselines: Establish normal operating ranges for key metrics (e.g., motor current, temperature). Machine learning models or statistical process control can then flag anomalies.
• Root Cause Triangulation: By correlating faults across logs, metrics, and traces, the system can infer root causes. Example: A spike in planning latency (metric) correlated with a PointCloud dropout log message and a trace showing the perception node timing out.
• Triggering Safe States: Diagnosed faults can trigger predefined recovery actions—from retrying a grasp to executing a safe stop—or escalate to a human operator. This is a key component of Functional Safety (FuSa).

Modern robotic systems are distributed networks of nodes. End-to-end tracing is essential to understand cross-node interactions.

• Context Propagation: Tools like ROS 2's tracetools inject unique trace IDs into message headers, allowing a single task (e.g., navigate_to_goal) to be followed across the perception, localization, planning, and control nodes.
• Visualizing Interaction Graphs: Traces generate graphs showing node dependencies, communication latency, and message flow. This reveals bottlenecks, such as a slow global planner delaying the entire pipeline.
• Integration with Middleware: This relies on the capabilities of the underlying middleware, such as ROS 2 built on the Data Distribution Service (DDS), which provides discoverable topics and services that form the natural boundaries for traces.

Telemetry is the automated collection and transmission of measurements and operational data from remote or inaccessible sources (like a robot in the field) to an IT system for monitoring. In robotics, this is the raw data stream that feeds observability.

• Examples: Motor current, joint encoder positions, battery voltage, internal temperature, network latency.
• Key Distinction: While observability is the property of the system, telemetry provides the data that makes observability possible. Effective observability requires high-fidelity, low-latency telemetry from all critical subsystems.

Logs are timestamped, immutable records of discrete events that occurred within a system, providing a chronological narrative of its operation. They are one of the three pillars of observability (alongside metrics and traces).

• Structured vs. Unstructured: Modern robotic systems use structured logs (e.g., JSON) for machine parsing, containing key-value pairs like {"event": "grasp_initiated", "object_id": "cup_23", "confidence": 0.92}.
• Purpose: Debugging complex, non-deterministic failures, auditing actions for safety compliance, and understanding the sequence of events leading to an error state.

Metrics are numerical measurements representing the state or performance of a system at a point in time, aggregated over intervals. They provide a quantitative, historical view of system health and are optimized for alerting and visualization.

• Types in Robotics:
  • Gauges: Instantaneous measurements (e.g., CPU usage %, available memory).
  • Counters: Monotonically increasing values (e.g., total tasks executed, communication errors).
  • Histograms: Statistical distributions (e.g., end-effector positioning error over 100 attempts, planning loop latency).
• Use Case: Creating dashboards for system vitals and triggering alerts when thresholds are breached (e.g., motor temperature > 80°C).

Distributed tracing is a method of observing requests as they propagate through a distributed system, like a robotic software stack. A trace is a directed acyclic graph of spans, where each span represents a unit of work (e.g., a function call, a ROS 2 node callback).

• Critical for Robotics: A single command (e.g., "pick up the block") triggers work across perception, planning, control, and actuator nodes. A trace visualizes the entire workflow, showing which component contributed to latency or failure.
• Implementation: Uses context propagation with unique trace IDs. Tools like OpenTelemetry provide standardized APIs for instrumenting ROS 2 nodes and other middleware.

Fault Detection and Diagnostics is an engineering discipline focused on identifying anomalies, errors, or failures in a system and determining their root cause. It is the active application of observability data.

• Detection: Using observability tools (metrics, logs) to identify that a system state has deviated from its expected norm (e.g., a LiDAR point cloud has sudden, persistent noise).
• Diagnostics: Analyzing traces, correlated logs, and historical metrics to isolate the faulty component (e.g., determining if the noise is from a dirty sensor lens, a loose connector, or a software driver bug).

A Digital Twin is a virtual, dynamic representation of a physical robotic system that mirrors its state, behavior, and context in real-time using sensor data and models. It is a powerful platform for observability and simulation.

• Observability Role: Acts as a unified, visual interface for system state. Engineers can observe the robot's estimated pose, sensor readings, planned path, and internal software state simultaneously in a simulated 3D environment.
• Proactive Analysis: Allows "what-if" scenarios and predictive maintenance by running the twin's models forward in time based on current telemetry.