Performance Baseline

The core of a baseline is a time-series dataset of Service Level Indicator (SLI) values collected during a period of known stable operation. This includes metrics like End-to-End Task Latency, Planning Success Rate, and Action Success Ratio. The dataset must be statistically significant, covering various operational conditions (e.g., different times of day, load levels) to represent the agent's "normal" behavior accurately. It is stored in a time-series database (e.g., Prometheus, InfluxDB) for querying and comparison.

A baseline transforms raw historical data into actionable reference boundaries. This involves calculating statistical measures like:

• Mean/Average: The central tendency of an SLI.
• Percentiles (P50, P95, P99): Critical for latency metrics to understand tail performance.
• Standard Deviation/Variance: Measures the dispersion of data points.
• Control Limits: Statistically derived upper and lower bounds (e.g., using 3-sigma rules) that define the expected range of normal variation. Current SLI values are compared against these bounds to flag anomalies.

A robust baseline is not a single number but a multi-dimensional profile. It includes metadata that segments performance by context, enabling precise comparisons. Key dimensions are:

• Agent Version/Deployment: To compare performance across releases.
• Input Complexity/Type: Baseline for simple queries vs. complex multi-step tasks.
• External Dependency Health: Performance correlated with the status of called APIs or data sources.
• User/Client Segment: Different baselines for internal vs. external user traffic.
• Time-of-Day/Day-of-Week: Capturing cyclical patterns in load and performance.

The process for creating a trusted baseline is formalized. It is not simply "last week's data." The protocol defines:

• Observation Window: A sufficient period (e.g., 7-14 days) of incident-free operation.
• Exclusion Criteria: Rules for filtering out data from known incidents, deployments, or maintenance windows that would skew the baseline.
• Stability Criteria: Quantitative checks to ensure the collected data represents a steady state (e.g., low variance in key SLIs).
• Approval Workflow: Designated engineers or SREs must validate and approve a new baseline before it is promoted for active use in monitoring.

The operational component that continuously compares live Agentic SLI streams against the established baseline. This engine:

• Calculates Delta/Deviation: Measures the difference between current values and baseline percentiles or means.
• Triggers Anomaly Detection: Uses algorithms (e.g., threshold breaches, statistical process control, machine learning models) to identify significant deviations.
• Generates Baseline-Aware Alerts: Alerts that reference the baseline (e.g., "P95 latency is 40% above baseline") provide immediate, actionable context for on-call engineers.

Baselines are versioned artifacts with a defined lifecycle. As agents evolve, their normal performance profile changes. Management includes:

• Baseline Versioning: Each approved baseline snapshot is tagged and immutable.
• Automated Re-baselining Triggers: Rules that initiate the creation of a new baseline after a significant agent deployment or when statistical drift is detected over time.
• Retention Policy: Historical baselines are retained for longitudinal analysis and audit.
• Canary Comparison: New agent versions are evaluated by comparing their canary deployment metrics against the production baseline to assess impact.

An Agentic SLI is a quantitative measure of a specific aspect of an autonomous agent's performance. It is the raw metric from which a baseline is established and against which SLOs are set.

• Examples: Planning Success Rate, End-to-End Task Latency, Action Success Ratio.
• Function: Provides the measurable data point for system health.
• Relationship to Baseline: The Performance Baseline is a historical record of normal SLI values.

An Agentic SLO is a target value or range for an Agentic SLI. It defines the acceptable level of service for the autonomous agent over a compliance period.

• Purpose: Creates a formal reliability target for engineering teams.
• Example: "Planning Success Rate ≥ 99.5% over a 30-day rolling window."
• Relationship to Baseline: The Performance Baseline informs realistic SLO targets. SLO violations are detected by comparing current SLI values against the target, with the baseline providing context for whether a deviation is anomalous.

An Error Budget is the allowable amount of time an autonomous agent system can fail to meet its SLOs within a defined period. It is calculated as (100% - SLO%) * Compliance Period.

• Function: Balances reliability with innovation velocity. Exhausting the budget should trigger a focus on stability over new features.
• Example: For a 99.5% monthly SLO, the error budget is 0.5% of the month (~3.6 hours).
• Relationship to Baseline: A stable Performance Baseline helps establish a predictable error consumption rate. Sudden increases in burn rate often indicate a deviation from the baseline.

Agentic Anomaly Detection refers to systems that identify deviations from normal operational patterns in agent behavior, decision-making, or performance metrics.

• Mechanisms: Uses statistical models, machine learning, or rule-based thresholds on SLI streams.
• Key Input: The Performance Baseline is the fundamental reference for defining "normal." Anomalies are signals that current SLI values have statistically diverged from this historical norm.
• Output: Triggers alerts or initiates automated remediation workflows.

A Canary Success Metric is a specific Agentic SLI or set of SLIs used to evaluate the health of a new agent version deployed to a small subset of traffic.

• Process: New version (canary) and baseline version (control) run concurrently. Their SLI performance is compared.
• Primary Comparison: The canary's SLI values are compared against two references: 1) the Performance Baseline of the stable system, and 2) the current SLIs of the control group.
• Decision Gate: If the canary's metrics remain within acceptable bounds of the baseline, the deployment proceeds.

A Composite SLI is a Service Level Indicator derived from mathematically combining two or more underlying Agentic SLIs into a unified score.

• Purpose: Measures complex, multifaceted aspects of agent performance like overall efficiency ((Task Completion Rate * Action Success Ratio) / Avg. Cost) or safety score.
• Calculation: Often a weighted formula (e.g., w1*SLI1 + w2*SLI2).
• Relationship to Baseline: A Performance Baseline must be established for the Composite SLI itself, as its behavior is an emergent property of the combined metrics.