Agent Tool Failure Injection Test Prompt

This prompt is for reliability engineers and AI quality engineers who need to verify that an agent handles tool failures gracefully before it reaches production. The job-to-be-done is generating a structured test plan that injects specific failure modes—timeouts, authentication errors, malformed responses, rate limits—into tool calls and defines the expected agent behavior for each scenario. You should use this prompt when you have a defined tool contract, a known set of failure modes you want to exercise, and a need for pass/fail criteria that can be automated in a CI pipeline or manual review gate.

The ideal user has access to the tool's API specification, error catalog, and the agent's system prompt or behavior policy. Required context includes the tool's name, its argument schema, the expected success response shape, and a list of failure modes to inject. You must also provide the agent's expected behavior policy—such as retry limits, fallback tools, escalation rules, and user communication requirements—so the generated test plan can define precise pass/fail criteria. Do not use this prompt for testing the tool implementation itself; it tests the agent's reaction to tool failures, not the tool's internal logic. It is also not a substitute for load testing, security penetration testing, or end-to-end integration tests that require live tool endpoints.

Before running this prompt, ensure you have a mock tool harness or simulation layer that can inject the specified failures without touching production systems. The generated test plan will assume you can control tool responses. After generating the plan, review each scenario for completeness against your operational runbooks—missing a timeout threshold or an idempotency requirement can cause the agent to loop, hallucinate, or silently drop work in production. Wire the pass/fail criteria into your eval framework so that every agent deployment is gated on these failure-mode tests.

This prompt template is the core of the failure injection test plan generator. It takes a tool contract, a set of failure modes to inject, and the agent's expected behavioral contract, then produces a structured test plan that reliability engineers can execute directly. The template is designed to be wired into a test harness where mocked tool responses are injected and the agent's actual behavior is compared against the plan's assertions.

textCopy
You are a reliability engineer designing failure injection tests for an AI agent that uses external tools.

Your task is to produce a structured test plan that injects specific failure modes into tool calls and defines the expected agent behavior for each failure scenario.

## INPUTS

[TOOL_CONTRACT]
[AGENT_BEHAVIOR_CONTRACT]
[FAILURE_MODES_TO_INJECT]
[CONSTRAINTS]

## INSTRUCTIONS

For each failure mode specified in [FAILURE_MODES_TO_INJECT], generate a test case with the following structure:

1. **Failure Injection**: Describe exactly what the mocked tool returns or does. Include:
   - The tool name and specific endpoint or function
   - The injected response (status code, error body, timeout duration, malformed payload)
   - Any preconditions required before injection (e.g., "agent must have completed authentication step first")

2. **Expected Agent Behavior**: Define what the agent must do when it receives this failure. Include:
   - Whether the agent should retry, fall back, escalate, or abort
   - The maximum number of retry attempts allowed
   - The expected user-facing message or system signal
   - Any state changes the agent should or should not make

3. **Pass/Fail Criteria**: Define observable, automatable assertions that determine whether the agent handled the failure correctly. Each criterion must be:
   - Binary (pass or fail, no partial credit)
   - Observable in logs, agent output, or system state
   - Independent of other test cases

4. **Risk Level**: Classify the test case as LOW, MEDIUM, HIGH, or CRITICAL based on:
   - Data integrity risk (could the agent corrupt state?)
   - User impact (would the user see incorrect results?)
   - Recovery difficulty (can the agent self-recover?)

## OUTPUT SCHEMA

Return a JSON object with this exact structure:

{
  "test_plan_id": "string",
  "generated_at": "ISO-8601 timestamp",
  "tool_under_test": "string (tool name from contract)",
  "failure_categories_covered": ["string"],
  "test_cases": [
    {
      "test_id": "string (unique, kebab-case)",
      "failure_mode": "string (e.g., timeout, auth_error, malformed_response, rate_limit, internal_error)",
      "risk_level": "LOW | MEDIUM | HIGH | CRITICAL",
      "injection": {
        "tool_name": "string",
        "mocked_response": {},
        "preconditions": ["string"]
      },
      "expected_behavior": {
        "action": "retry | fallback | escalate | abort | ignore",
        "max_retries": "integer or null",
        "fallback_tool": "string or null",
        "user_signal": "string (expected message or indicator)",
        "state_changes": "string (description of allowed or forbidden state mutations)"
      },
      "pass_criteria": [
        {
          "criterion_id": "string",
          "description": "string (observable assertion)",
          "check_method": "log_assertion | output_assertion | state_assertion | metric_assertion",
          "expected_value": "string"
        }
      ]
    }
  ],
  "coverage_gaps": ["string (failure modes not covered and why)"],
  "execution_order": ["string (test_id sequence with dependency notes)"]
}

## CONSTRAINTS

- Do not invent failure modes not listed in [FAILURE_MODES_TO_INJECT]
- Every pass criterion must be automatable—no manual judgment required
- If a failure mode cannot be safely tested (e.g., destructive side effects), flag it in coverage_gaps with the reason
- Respect [CONSTRAINTS] if provided (e.g., "no production endpoints", "read-only tools only")
- If [AGENT_BEHAVIOR_CONTRACT] is incomplete for a failure mode, note the ambiguity and suggest the contract gap

To adapt this template, replace each square-bracket placeholder with concrete inputs. [TOOL_CONTRACT] should contain the full tool definition including endpoints, argument schemas, and expected success responses. [AGENT_BEHAVIOR_CONTRACT] should describe the agent's documented retry policy, fallback chain, escalation path, and user communication rules. [FAILURE_MODES_TO_INJECT] should be a specific list drawn from your production incident history—timeouts, 429 rate limits, 503 service unavailable, malformed JSON responses, and authentication expiry are the minimum starting set. [CONSTRAINTS] is optional but critical for safety: always include "no production endpoints, no destructive writes, use mocked responses only" when generating tests that will be executed automatically. After copying the template, validate that the output JSON matches the schema before feeding it into your test runner.

This prompt is designed to be the planning stage of an automated reliability pipeline, not a one-off chat interaction. The output is a structured test plan (JSON) that should be parsed by a test runner or orchestration engine. Do not treat the generated plan as a final artifact—it must pass through a validation layer that checks for schema completeness, scenario coverage against your tool registry, and safety constraints before any failure injection is executed against a live or staging agent.

Wire the prompt into a three-stage harness: (1) Generation—call the model with the prompt template, supplying your tool contract schemas, known failure catalog, and risk tolerance. Use a model with strong structured output support (GPT-4o, Claude 3.5 Sonnet, or equivalent) with response_format set to json_schema if available. (2) Validation—run the output through a deterministic validator that checks: every scenario has a unique ID, every injected failure maps to a tool in your registry, pass/fail criteria are non-empty, and no destructive mutations (DELETE, DROP, production-write) are present unless explicitly allowed. Reject and retry with error feedback if validation fails. (3) Execution—feed validated scenarios one at a time into your test harness, which mocks the tool at the boundary, invokes the agent, and evaluates behavior against the pass/fail criteria. Log every scenario outcome with the scenario ID, actual agent behavior, and a pass/fail determination.

For high-risk production agents (those with write access to databases, financial systems, or user data), add a human approval gate between validation and execution. The validated test plan should be surfaced in a review UI showing: which tools are targeted, what failure modes are injected, and whether any scenario could cause side effects in connected staging environments. Only after explicit approval should the harness proceed. For regression use, store validated test plans alongside your agent version so that every agent release re-runs the same failure injection suite. Track pass rates over time and flag regressions where an agent previously handled a failure mode correctly but now hallucinates, loops, or times out.