Alerting Rule

The Condition is the logical expression that evaluates one or more Agentic SLIs. The Threshold is the specific value that, when breached, triggers the alert.

• Example Condition: planning_success_rate < 95% over a 5-minute window.
• Threshold Types: Static (e.g., latency > 2s), dynamic (e.g., deviation from a rolling baseline), or composite (evaluating multiple SLIs).
• For Agentic Systems: Conditions often monitor SLIs like Hallucination Rate, Self-Correction Success Rate, or Guardrail Compliance Rate to detect reasoning failures.

The Evaluation Window is the time range of data (e.g., 'last 5 minutes') analyzed by the condition. The Frequency is how often the rule is evaluated against new telemetry.

• Purpose: Prevents noise from transient spikes. A short window (e.g., 1 min) catches rapid failures; a longer window (e.g., 15 min) identifies sustained degradation.
• Critical for Agents: Agent behavior can be bursty. Rules for End-to-End Task Latency may use a 1-minute window, while rules for Cost Per Successful Task may use an hourly window to smooth variance.
• Common Pattern: eval_interval: 30s, window: 5m means the rule checks the last 5 minutes of data every 30 seconds.

Severity (e.g., Critical, Warning, Info) indicates the impact of the breach. Labels are key-value pairs (e.g., agent_id=planner_01, slo=planning-success) attached to the alert for routing and context.

• Severity Mapping: A breached SLO Burn Rate might be Critical; a minor Redundant Action Ratio increase might be Warning.
• Label Use Cases:
  • Routing: team=agent-sre routes to the correct on-call group.
  • Aggregation: slo_violation=true groups all related alerts.
  • Context: failed_tool=stock_api provides immediate diagnostic data.

Defines where and how the alert is sent (e.g., PagerDuty, Slack, email). Routing logic uses alert labels to direct notifications.

• Channel Examples: PagerDuty for severity=critical, Slack channel for severity=warning.
• Agent-Specific Channels: Alerts for Hallucination Rate may route to a dedicated #ai-ethics-review channel, while Tool Call failures route to the API engineering team.
• Escalation Policies: Can be defined based on alert duration (e.g., if unresolved for 10 minutes, escalate).

The for clause introduces a delay, requiring the condition to be true for a sustained period (e.g., for: 2m) before firing, reducing flapping. Grouping aggregates alerts by labels to prevent notification storms.

• Example: planning_success_rate < 95% for 3m.
• Grouping Example: Alerts for 100 agent instances can be grouped by agent_type, sending one summary alert per type instead of 100 individual alerts.
• Agent Utility: Essential for stateful agents where a single failed planning cycle may not indicate a systemic issue.

Annotations are human-readable details (summary, description) attached to the alert. Runbooks are linked diagnostic and mitigation procedures.

• Annotation Fields:
  • summary: 'Planning SLO Violation: Success Rate at 92%'.
  • description: 'The planner agent is failing to decompose complex logistics tasks. Check recent memory ingestion logs.'
• Runbook Links: A URL to a playbook for troubleshooting specific Agentic SLI breaches, such as steps to restart a Reasoning loop or validate Tool Calling credentials.
• Purpose: Enables faster Mean Time To Resolution (MTTR) by providing immediate context to responders.

An Agentic SLI is the quantitative measurement that an alerting rule monitors. It is a specific, measurable attribute of an autonomous agent's performance, such as Planning Success Rate or End-to-End Task Latency. An alerting rule is defined on an SLI to detect when its value breaches a threshold.

• Examples: Task Completion Rate, Hallucination Rate, Cost Per Successful Task.
• Purpose: Provides the raw, objective data signal for health assessment.

An Agentic SLO is the business target for an SLI. It defines the acceptable performance level (e.g., "Planning Success Rate must be ≥ 99.5% over 30 days"). Alerting rules are configured to fire when an SLI's value indicates a risk of SLO violation or has already breached the objective.

• Relationship to Alerting: SLOs inform the severity and thresholds of alerting rules.
• Error Budget: The allowable deviation from an SLO; alerting on burn rate is a proactive strategy.

The Error Budget quantifies the acceptable unreliability for a service, derived from its SLOs. For autonomous agents, it's the allowable time or number of failures before an SLO is violated. Alerting rules can be set on SLO Burn Rate—the speed at which the error budget is being consumed—to trigger warnings long before a final breach occurs.

• Proactive Alerting: A fast burn rate triggers alerts for investigation, enabling preventative action.
• Balances Innovation & Reliability: Guides deployment and change management decisions.

A Performance Baseline is a historical record of normal SLI values established during stable operation. Effective alerting rules use dynamic thresholds informed by baselines, rather than static values, to account for normal cyclical patterns (e.g., lower throughput at night). This reduces false positives and helps detect true anomalies.

• Use Case: An alert triggers if latency deviates by more than 3 standard deviations from the 7-day baseline.
• Foundation for Anomaly Detection: Essential for machine learning-based alerting systems.

Agentic Anomaly Detection refers to systems that automatically identify deviations from normal behavior in agent metrics or logs. While a basic alerting rule uses a fixed threshold, anomaly detection uses statistical or ML models to define "normal" dynamically. Their outputs can themselves be configured as composite SLIs to trigger alerts.

• Advanced Alerting: Detects subtle, multi-dimensional shifts that simple thresholds miss.
• Examples: Unusual planning loop iterations, atypical tool call sequences, or emergent coordination failures.

A Composite SLI is a single metric synthesized from multiple underlying SLIs (e.g., an efficiency score combining Cost Per Task and Redundant Action Ratio). Alerting rules can be defined on these higher-level indicators to represent complex service health concepts. This reduces alert fatigue by consolidating signals.

• Simplifies Monitoring: One alert on a composite SLI can replace several on individual metrics.
• Represents Business Health: Can directly reflect user experience or operational efficiency.