Golden Signals

Latency measures the time required to service a request, typically expressed as a duration (e.g., milliseconds). It is the primary indicator of user-perceived performance.

• Types: Distinguish between success latency (time for successful requests) and failure latency (time for failed requests, which can be very fast).
• Monitoring: Track percentiles (p50, p95, p99) rather than averages to understand tail latency and outliers.
• Agent Context: In a multi-agent system, measure the end-to-end latency of a user query as it flows through the orchestration workflow, as well as the internal latency of individual agent reasoning and tool calls.

Traffic quantifies the demand placed on a system, measuring how much work it is doing. It is a direct indicator of usage and load.

• Metrics: Common measures include requests per second (RPS), transactions per second (TPS), or the volume of data processed.
• Agent Context: For orchestration, traffic can be measured as the number of tasks dispatched per second, messages exchanged between agents, or the rate of agent invocations. A sudden drop in traffic can indicate a failure in upstream systems or agent registration/discovery issues.

Errors track the rate of requests that fail, either explicitly (e.g., HTTP 5xx status codes) or implicitly (e.g., returning incorrect or partial results).

• Explicit vs. Implicit: Monitor both hard failures (timeouts, crashes) and soft failures (business logic errors, incorrect agent outputs).
• Agent Context: In agentic systems, errors include failed tool executions, LLM provider API errors, agent reasoning failures, protocol violations, and deadlock scenarios in coordination. Errors should be tracked per agent type and per workflow step.

Saturation measures how "full" a system resource is, indicating the pressure on a constrained resource before performance degrades. It is a forward-looking signal of impending failure.

• Resources: Common saturation points include CPU load, memory utilization, I/O throughput, and queue depths.
• Agent Context: For orchestration, monitor the saturation of shared resources like vector database connections, LLM API rate limits, message broker queues, and the concurrent task capacity of individual agents. High saturation often precedes increased latency and errors.

In a multi-agent system, the Golden Signals must be interpreted through the lens of distributed, concurrent workflows.

• Latency: Trace a single user request across the agent call graph to identify slow agents or communication bottlenecks.
• Traffic: Correlate spikes in task initiation with downstream saturation of specialized agents (e.g., a research agent).
• Errors: Use structured logging to tag errors with the specific agent, tool, and workflow ID for root cause analysis.
• Saturation: Monitor the queue depth in the orchestration workflow engine; a growing backlog indicates the system cannot keep up with incoming traffic.

The Golden Signals are most powerful when used to define and track Service Level Objectives (SLOs).

• SLO Definition: An SLO might state "99% of agent workflow completions shall have a latency under 2 seconds."
• Error Budget: The error budget is the allowable amount of SLO violation (e.g., 1% of requests can be slow). Exhausting this budget should trigger a focus on stability over new features.
• Proactive Management: By monitoring these signals against SLOs, teams can perform canary analysis on new agent deployments or conduct chaos engineering experiments to test system resilience before incidents occur.

Distributed tracing is a method for profiling requests as they propagate through a distributed system. In a multi-agent context, it tracks a single user request or orchestration task as it triggers a cascade of agent calls, message passes, and tool executions.

• Creates an end-to-end trace composed of individual spans for each operation.
• Essential for diagnosing high latency (a Golden Signal) by pinpointing the slowest agent or communication link in a workflow.
• Tools like OpenTelemetry provide standardized instrumentation for tracing.

OpenTelemetry is a vendor-neutral, open-source observability framework. It provides APIs, libraries, and agents to instrument your multi-agent system, generating a unified stream of traces, metrics, and logs.

• Metrics from OTel can directly feed the Golden Signals (e.g., request counts for traffic, error rates for errors).
• Traces provide the detailed context for those metrics, showing why an error occurred or latency spiked.
• Acts as the foundational data collection layer for an observability pipeline, decoupling instrumentation from analysis tools.

A Service Level Objective is a target for a specific reliability metric over a time window. SLOs operationalize the Golden Signals into business agreements.

• Example: "Agent workflow success rate ≥ 99.9% over 30 days." This directly uses the error signal.
• Another: "95% of agent responses completed in < 2 seconds." This uses the latency signal.
• The difference between the SLO (e.g., 99.9%) and 100% is the Error Budget, which quantifies how much unreliability can be tolerated for new feature releases.

Structured logging is the practice of writing log events in a machine-parsable format (like JSON) with explicit key-value pairs, instead of plain text. This is critical for debugging autonomous agents.

• Enables powerful filtering and aggregation: { "agent": "planner", "task_id": "abc123", "decision": "decompose", "confidence": 0.87 }
• When an error Golden Signal fires, structured logs from the failing agent provide immediate context without complex parsing.
• Feeds efficiently into Centralized Log Aggregation systems (e.g., Loki, Elasticsearch) for correlation with traces and metrics.

Health checks are automated probes that verify the operational readiness of a software component. For agent orchestration, they test both individual agents and the coordination framework.

• Liveness Probe: Is the agent process running? If not, the orchestrator may restart it.
• Readiness Probe: Is the agent initialized, connected to its memory (e.g., vector DB), and able to process requests? If not, traffic is routed away.
• A failing health check is a direct, binary indicator of saturation (agent is dead) or errors (agent is malfunctioning).

An agent call graph is a visual or data representation mapping the interactions between agents during a specific task execution. It is the topological output of distributed tracing for a multi-agent system.

• Shows the sequence of agent activations, message flows, and tool calls.
• Critical for understanding complex workflows, identifying bottlenecks (contributing to latency), and spotting circular dependencies or agent conflicts.
• Acts as a blueprint for the orchestration workflow engine, helping to validate that the executed path matches the designed plan.