Result Accuracy

Result Accuracy is formally defined as the percentage of tasks where an autonomous agent's final output is deemed correct against a ground truth or human evaluation. It is calculated as (Number of Correct Tasks / Total Tasks Attempted) * 100. This SLI directly measures the agent's primary functional objective: producing valid results. Unlike simpler classification accuracy, it often applies to complex, multi-step outputs like code generation, report synthesis, or multi-document analysis.

Accurate measurement is fundamentally dependent on a reliable ground truth or gold standard dataset. For deterministic tasks (e.g., code execution, data extraction), ground truth can be automated. For subjective or creative tasks, it requires human-in-the-loop evaluation or consensus from multiple expert reviewers. The quality and consistency of the ground truth directly limit the validity of the Result Accuracy metric. Ambiguous or incomplete reference data leads to unreliable SLI values.

Accuracy can be measured at different levels of granularity within an agent's workflow:

• End-to-End Task Accuracy: The final, aggregated output is correct.
• Sub-task or Step Accuracy: Individual components of a plan (e.g., a single API call result, a reasoning step) are correct. Monitoring at the sub-task level provides deeper diagnostic insight, helping to isolate whether failures originate in planning, tool execution, or synthesis. A high sub-task accuracy with low end-to-end accuracy may indicate integration or logic errors in the agent's orchestration layer.

Result Accuracy does not exist in isolation and must be interpreted alongside complementary SLIs:

• Hallucination Rate: Measures factual unsupported content; a high rate directly degrades Result Accuracy.
• Planning Success Rate: An agent that fails to plan correctly cannot produce an accurate result.
• Action Success Ratio: Failed tool executions prevent accurate task completion.
• Self-Correction Success Rate: An agent's ability to fix its own errors can recover accuracy. A holistic view requires tracking this constellation of SLIs to understand the root cause of accuracy failures.

Establishing Result Accuracy scales from manual to fully automated methods:

• Human Evaluation: Gold standard but expensive, slow, and can suffer from rater bias. Essential for establishing initial baselines and evaluating subjective tasks.
• Rule-Based Checks: For tasks with strict formatting or deterministic outputs (e.g., "extract the invoice total"), automated validation against schemas or regular expressions is possible.
• Model-Based Evaluation: Using a secondary, often more powerful or specialized LLM as a judge to score outputs. This introduces its own evaluation bias and cost but enables scale. Most production systems use a hybrid approach.

A Result Accuracy SLO sets the target acceptable level, e.g., "99% of agent-generated financial summaries must be factually correct." The Error Budget (e.g., 1% incorrect outputs per quarter) quantifies allowable failure. This budget is consumed by incidents of low accuracy. Monitoring the SLO Burn Rate—how quickly the error budget is being used—is critical for release governance. A high burn rate for Result Accuracy may halt deployments of new agent versions until the root cause is addressed.

An Agentic SLI is a quantitative measure of a specific aspect of an autonomous agent's performance. Unlike traditional SLIs, they are tailored to cognitive and behavioral metrics intrinsic to AI agents.

• Examples: Planning Success Rate, Task Completion Rate, Result Accuracy, End-to-End Task Latency.
• Purpose: Provides the raw, measurable data points used to assess an agent's operational health and effectiveness.

An Agentic SLO is a target value or range for an Agentic Service Level Indicator (SLI). It defines the acceptable level of performance for an autonomous agent system over a specified period.

• Structure: "Result Accuracy must be ≥ 99.5% over a 30-day rolling window."
• Function: SLOs, paired with an Error Budget, create a data-driven framework for balancing reliability with the velocity of new agent deployments and feature releases.

An Automated Evaluation Score is a metric generated by a rule-based or model-based system to assess the quality of an autonomous agent's output without human intervention. It is a primary method for calculating SLIs like Result Accuracy at scale.

• Mechanisms: Can use rule-based checkers (e.g., code syntax validation, regex pattern matching), model-based graders (a smaller LLM judging a larger one), or canonical answer comparison.
• Key Challenge: Designing evaluation systems that are themselves accurate, unbiased, and resistant to adversarial manipulation.

Hallucination Rate is an Agentic SLI that quantifies the frequency with which an autonomous agent generates factually incorrect or unsupported information. It is a critical inverse metric to Result Accuracy, specifically focusing on the fabrication of information.

• Measurement: Often calculated as the percentage of agent outputs containing unverifiable or contradicted statements.
• Mitigation: Reduced through techniques like Retrieval-Augmented Generation (RAG), guardrails, and improved prompt architecture.

An Error Budget is the allowable amount of time an autonomous agent system can fail to meet its Service Level Objectives (SLOs) within a defined compliance period. It is derived directly from SLO targets.

• Calculation: If the SLO is 99.9% monthly availability, the error budget is 0.1% of the month (~43 minutes).
• Operational Use: This budget is consumed by outages or performance degradations. Exhausting the budget should trigger a freeze on risky changes, focusing engineering effort on improving reliability.

A Performance Baseline is a historical record of normal Agentic SLI values for an autonomous agent, established during a period of known-stable operation. It serves as the reference point for detecting performance degradation.

• Establishment: Created by measuring SLIs like Result Accuracy and Task Latency over an initial calibration period post-deployment.
• Application: Used for anomaly detection (deviations from baseline trigger alerts) and for evaluating the impact of new agent versions in canary deployments.