Acceptance Criteria

Each criterion must be a binary condition that yields a clear pass/fail result. Vague language like "user-friendly" or "fast" is replaced with quantifiable, verifiable statements. This enables direct translation into automated test cases within a verification pipeline.

• Example (Bad): "The system should respond quickly."
• Example (Good): "The search API endpoint must return a response within 200 milliseconds for 95% of requests under a load of 100 queries per second."

Criteria should be written from the user's perspective, describing the value delivered. They follow the INVEST mnemonic for quality user stories: Independent, Negotiable, Valuable, Estimable, Small, Testable. This ensures the feature delivers tangible business or user value, not just technical completeness.

• Focus on Outcome: Criteria define what the user achieves, not how the system implements it.
• Example: "As a customer, I can apply a discount code at checkout so that I pay the reduced price" rather than "The system shall validate the promo_code field against the database."

A comprehensive set of criteria must define all conditions for satisfaction, including happy paths, alternative flows, and error handling. This involves explicitly stating preconditions, postconditions, and business rules. Effective criteria proactively address edge cases and boundary conditions to prevent ambiguous outcomes.

• Example Coverage: For a login feature, criteria would cover successful login, incorrect password, nonexistent user, account locked, and network timeout scenarios.
• Boundary Testing: "The quantity selector must accept values from 1 to 99 inclusive. Entering 0 or 100 displays an error message."

Each individual criterion should be atomic, representing a single, indivisible requirement. This prevents "partial pass" scenarios and simplifies test mapping. Overly complex criteria are decomposed. Conciseness ensures they are easily understood by all stakeholders—developers, testers, and product owners.

• Atomic Example: Instead of "The user can save and submit the form," split into: "1. The 'Save Draft' button persists form data. 2. The 'Submit' button validates all fields and posts data."
• Avoid Conjunctions: Watch for "and" or "or" within a criterion, as they often signal a need to split.

Acceptance criteria are the primary input to a team's Definition of Done (DoD). The DoD is a checklist of activities required to consider a work item complete (e.g., code reviewed, tests passed, documented). Criteria fulfillment is the central item on this checklist. This alignment ensures that "done" means accepted by the product owner, not just that code is merged.

• Pipeline Integration: In MLOps and agentic systems, this means criteria are encoded into automated validation suites that must pass before a model or agent deployment is considered complete.

These criteria define the quantitative performance thresholds a model must achieve before deployment.

• Performance Metric Thresholds: The model must achieve an F1 score of >= 0.92 and a precision of >= 0.95 on the held-out golden dataset.
• Latency Constraints: The model's p99 inference latency must be < 100 milliseconds when served on the target hardware.
• Fairness Guardrails: The model's false positive rate must not vary by more than 5% across all protected demographic subgroups defined in the training data.
• Resource Limits: The model's memory footprint must not exceed 2 GB when loaded into the inference server.

These criteria validate the functional correctness, safety, and reliability of an autonomous agent's outputs and behaviors.

• Output Format Compliance: The agent's response must be a valid JSON object matching the specified schema, with no extraneous text.
• Tool Calling Accuracy: When invoking an external API, the agent must construct the HTTP request with 100% correct parameter mapping as defined in the OpenAPI specification.
• Hallucination Prevention: For factual queries, 100% of cited information must be retrievable and verifiable from the provided context window or connected knowledge base.
• Recursive Error Handling: If an initial tool call fails, the agent must execute at least one, but no more than three, corrective action planning cycles before escalating to a fallback handler.

These criteria ensure coordinated, fault-tolerant behavior across a system of interacting autonomous agents.

• Orchestration Protocol Adherence: All inter-agent messages must conform to the defined agent communication language (ACL) and be logged with a unique transaction ID.
• Conflict Resolution: In scenarios of resource contention, the system must resolve the conflict using the designated strategy (e.g., priority-based, round-robin) within 5 seconds.
• Cascade Failure Prevention: The implementation of circuit breaker patterns must prevent a single agent failure from causing > 10% of the agent fleet to enter a failed state.
• Collective Goal Satisfaction: The multi-agent system must achieve the specified global objective (e.g., 'optimize warehouse pick path') with a solution cost within 15% of the simulated optimum.

These criteria guarantee the quality, timeliness, and integrity of data flowing into AI systems.

• Data Freshness SLO: All batch feature tables must be updated within 15 minutes of the scheduled execution time, 99.9% of the time.
• Schema Validation: 100% of ingested records must pass static analysis against the registered Avro or Protobuf schema; invalid records are routed to a dead-letter queue.
• Statistical Integrity: The mean and standard deviation of key numerical features in the production pipeline must not drift by more than 3 standard deviations from the training set statistics, as monitored by data drift detection.
• Lineage Completeness: Every feature served for inference must have complete, queryable lineage tracing back to its raw source, including all transformation steps.

These criteria are non-negotiable constraints designed to enforce ethical, legal, and operational boundaries.

• Content Moderation: The system must filter and block any output containing personally identifiable information (PII) with a recall of 1.0 (100%).
• Regulatory Adherence: All automated decision outputs must include the required legal disclosures as specified by jurisdiction (e.g., EU AI Act 'high-risk' system explanations).
• Adversarial Robustness: The system must maintain its core functionality when subjected to a suite of fuzzing tests, including prompt injection attempts, without leaking system prompts or internal logic.
• Resource Exhaustion Limits: The agent must halt execution and log an alert if a single task consumes more than its allocated budget of 10 LLM tokens or 5 tool calls.

These criteria validate the operational readiness and monitoring capabilities of the AI system in production.

• Canary Deployment Success: The new model version must outperform the baseline in the canary deployment environment on primary metrics for 24 hours with no critical alerts.
• Telemetry Coverage: 100% of agent actions, tool calls, and LLM requests must emit structured logs with fields for confidence scoring, latency, and a unique trace ID.
• Rollback Triggers: Automated rollback strategies must be invoked within 2 minutes if the system's aggregate error rate increases by >5% or if a circuit breaker is tripped.
• Health Check Pass: All agentic health checks, including connectivity to dependent vector databases and APIs, must return a 'healthy' status before the service is added to the load balancer pool.

A test harness is a collection of software, test data, and configuration used to execute automated tests and report on their outcomes. In the context of autonomous agents, it provides the execution environment for systematically validating outputs against acceptance criteria.

• Key Components: Test runners, mock APIs, data fixtures, and reporting dashboards.
• Primary Function: Automates the execution of a regression suite and integration tests to ensure new agent behaviors do not break existing functionality.
• Example: A harness that programmatically feeds a set of user queries to an agent, captures its tool calls and final answers, and compares them to expected results defined in a golden dataset.

A golden dataset is a curated, high-quality reference dataset used as a source of truth for validating model outputs and system behavior. It serves as the definitive benchmark (ground truth) against which an agent's performance is measured.

• Characteristics: Manually verified, version-controlled, and representative of edge cases.
• Role in Validation: Each entry in a golden dataset pairs an input with the correct, accepted output, forming the concrete basis for acceptance criteria.
• Usage: Used in smoke tests and comprehensive evaluation suites to detect data drift and concept drift by comparing live agent outputs to these canonical examples.

A guardrail is a software mechanism or policy designed to constrain a system's behavior to prevent undesirable, unsafe, or non-compliant outputs. While acceptance criteria define what is correct, guardrails enforce what is not allowed.

• Implementation Types: Can be pre-processing filters, post-output classifiers, or real-time monitoring rules.
• Function: Acts as a safety net to catch violations that may pass basic functional checks, such as generating harmful content, leaking sensitive data, or making unsubstantiated claims.
• Relation to Criteria: Often implemented as automated checks within a validation pipeline, working alongside acceptance tests to ensure comprehensive output safety and quality.

Confidence scoring is the process of quantifying and assigning probabilistic measures of certainty or reliability to an agent's generated results. It provides a meta-evaluation of whether an output is likely to meet acceptance criteria before formal validation.

• Methods: Can be derived from model logits, self-evaluation prompts, or ensemble agreement.
• Operational Use: Low-confidence scores can trigger iterative refinement protocols, dynamic prompt correction, or a human-in-the-loop review, creating a tiered validation system.
• Metric Integration: Scores are critical for agentic self-evaluation and can be monitored over time to detect performance degradation.

Property-based testing is a software testing methodology where tests verify that a function's output satisfies general logical properties for a wide range of automatically generated inputs. It complements example-based acceptance criteria by testing for invariant rules.

• Core Principle: Instead of testing specific input-output pairs, it defines properties (e.g., 'output is always valid JSON', 'response length never exceeds 500 characters') that must hold true for all inputs.
• Advantage: Excellent for discovering edge cases and ensuring robustness, as the test framework (test harness) generates hundreds of random or semi-random inputs.
• Application: Used to validate that an agent's output formatting, data types, and business logic invariants are universally maintained.

Shadow mode is a deployment technique where a new model or system processes live traffic in parallel with the production system, but its outputs are not used to affect user decisions. It is a critical validation step before accepting a new agent version.

• Purpose: To gather performance data on the new system under real-world conditions without risk. Outputs are logged and evaluated against acceptance criteria and the incumbent system's results.
• Validation Role: Enables comparison of key metrics (precision, recall, latency) and detection of anomalies or regressions before a canary deployment.
• Outcome: Provides empirical evidence that the new agent's outputs consistently meet the required standards, informing the final go/no-go decision for launch.