Golden Signals

Latency measures the time it takes to service a request. It is the primary indicator of user-perceived performance. Monitoring requires distinguishing between successful and failed request latencies, as a failed request (e.g., returning an HTTP 500 error) often completes quickly but provides no value.

• Key Insight: A fast error is still an error. Always track latency for successful and failed requests separately.
• Example: P95 latency for successful API calls should be under 200ms, while failed calls may show a P99 of 50ms.
• In Fleet Context: This translates to the time between a central orchestrator issuing a command (e.g., 'navigate to point B') and receiving acknowledgment of task completion from the agent.

Traffic quantifies how much demand is being placed on your system. It is a measure of load. The specific metric depends on the system type.

• For Web Services: HTTP requests per second.
• For Audio Streaming: Concurrent sessions or network I/O rate.
• For Databases: Queries per second or transactions per second.
• In Fleet Context: This is the volume of work. Metrics include:
  • Tasks assigned per minute to the fleet.
  • Total distance traveled by all agents per hour.
  • Messages per second on the inter-agent communication bus.

Monitoring traffic is essential for capacity planning and auto-scaling decisions.

Errors measure the rate of requests that fail. Failures can be explicit (e.g., HTTP 5xx status codes), implicit (e.g., an HTTP 200 response with invalid or empty content), or policy-based (e.g., a request that took too long, violating an SLO).

• Key Insight: Track both volume (errors per second) and ratio (error rate as a percentage of total traffic).
• In Fleet Context: Errors are multifaceted:
  • Agent Failures: Hardware faults, software crashes, liveness probe failures.
  • Task Failures: Navigation aborts, pick-up failures, communication timeouts.
  • Degraded Performance: A successful task that took significantly longer than expected can be considered a 'soft' error against an SLO.

Saturation measures how 'full' your system is. It indicates the utilization of a constrained resource. A system can be at 100% traffic capacity long before it reaches 100% saturation, which is the point where performance degrades.

• Resources to Monitor: CPU, memory, disk I/O, network bandwidth, and queue lengths.
• Critical Threshold: Performance degrades non-linearly as saturation approaches 100%.
• In Fleet Context: Saturation is critical for physical systems:
  • Agent-Level: Battery State of Charge (SoC), compute load, memory usage.
  • Fleet-Wide: Percentage of agents currently busy, charging station queue length, congestion in high-traffic zones (spatial saturation).
  • Infrastructure: Network bandwidth saturation in the warehouse.

In Heterogeneous Fleet Orchestration, the Golden Signals map directly to operational vitals, providing a universal health framework for mixed fleets of AMRs and manual vehicles.

• Latency → Task completion time & command-response cycles.
• Traffic → Total work volume (tasks/km/messages).
• Errors → Agent failures, task exceptions, and SLO violations.
• Saturation → Battery levels, zone congestion, and compute load.

This framework allows Site Managers and DevOps Engineers to move from monitoring individual agent logs to managing the health of the workflow itself. A spike in latency with high traffic and rising saturation may indicate fleet overload, while a high error rate with normal traffic could point to a systemic software bug.

While the classic four signals are sufficient for most services, some SRE practices advocate for a fifth signal: Tickets or Human Intervention.

• Definition: The volume of manual work generated for operators by the system. This measures toil.
• Purpose: A system with good latency, traffic, error, and saturation metrics can still be unhealthy if it requires constant manual intervention to function.
• In Fleet Context: This is the number of exception tickets requiring a human operator's attention.
  • Examples: An agent requiring manual unstucking, a task needing re-assignment, a hardware fault needing physical inspection.
• Goal: A healthy, autonomous fleet should minimize this signal through robust exception handling frameworks and graceful degradation.

A composite, often weighted, numerical value that summarizes the overall operational status of an agent or system, derived from multiple underlying Golden Signals and diagnostic checks.

• Calculation: Aggregates metrics like latency percentiles, error rates, battery saturation (State of Charge), and liveness probe results.
• Purpose: Provides a single, at-a-glance indicator of fleet or agent health for dashboards and automated alerting.
• Implementation: Requires careful normalization and weighting of disparate signal types (e.g., a dead battery vs. high latency).

A system design principle where an agent or service maintains partial, reduced functionality in the face of partial failures, which is monitored via Errors and Saturation.

• Objective: To avoid a complete system failure (0% availability) when a dependency fails or resources are saturated.
• Example: An Autonomous Mobile Robot (AMR) that loses its central tasking server might switch to a pre-cached patrol route or safe-hold position.
• Monitoring: Signals a shift from normal operation to a degraded state, which should be reflected in altered traffic patterns and potentially higher error rates for certain functions.

An algorithm used to space out retries of a failed operation, directly managing Traffic and Saturation to aid recovery.

• Process: After a failure (error), the client waits for a short interval before retrying. Each subsequent failure doubles the wait time (e.g., 1s, 2s, 4s, 8s).
• Purpose: Prevents a 'retry storm' that can overwhelm a failing service (saturating it) and gives the struggling system time to recover.
• Fleet Context: Used by agents retrying network calls, task submissions, or status updates to the orchestration middleware.

A key operational metric that measures the average time required to repair a failed component or agent and restore it to service. It is the human and systemic response to signals indicating failure.

• Relationship to Golden Signals: A high Error rate or a Saturation alert (e.g., 0% battery) triggers an incident. MTTR measures the efficiency of the response, from detection to resolution.
• Components: Includes time for alerting, diagnosis (aided by telemetry), repair action (e.g., OTA update, physical intervention), and verification.
• Goal: Reducing MTTR is often more impactful than increasing MTBF (Mean Time Between Failures) for overall system availability.