Agent Tool Retry Logic Test Prompt

This prompt is designed for reliability engineers and AI quality engineers who need to systematically validate an agent's retry behavior before it reaches production. The job-to-be-done is generating a structured test plan that exercises both transient and persistent tool failures, defining the expected retry strategy—including maximum attempts, backoff policy, idempotency handling, and escalation after exhaustion. You should use this prompt when you have a defined tool contract and need to verify that your agent's retry logic handles failure modes correctly rather than looping infinitely, abandoning tasks silently, or violating idempotency guarantees. The ideal user has access to the tool's API specification, understands the agent's retry configuration, and needs a repeatable test artifact that can be shared across the engineering and QA teams.

This prompt is not a substitute for integration tests that exercise real tool endpoints, nor is it designed for testing the tool's own internal retry behavior. It focuses exclusively on the agent's client-side retry logic—the decisions the agent makes when a tool call fails. You should avoid using this prompt when the agent has no configurable retry policy, when the tool is stateless and idempotent by default, or when you are testing the tool provider's infrastructure rather than your agent's resilience. The prompt assumes you can provide a tool contract that includes the endpoint signature, expected error codes, idempotency key semantics, and any documented retry-after headers or rate-limit responses. Without this context, the generated test plan will be generic and less actionable.

Before running this prompt, gather the tool's OpenAPI spec or equivalent contract documentation, the agent's current retry configuration (max retries, backoff multiplier, jitter settings, timeout thresholds), and any existing incident reports where retry behavior caused production issues. The output test plan should be treated as a living document—update it when the tool contract changes, when the agent's retry policy is tuned, or when new failure modes are discovered in production. If the tool performs irreversible side effects (writes, sends, deletes), require human review of the test plan before execution and ensure tests run against a sandbox or staging environment, never against production endpoints.

This prompt template generates a comprehensive test plan for validating an agent's retry logic against a specific tool. It takes your tool contract, failure modes, and retry policy as inputs and produces structured test scenarios with clear pass/fail criteria. The output is designed to be directly consumed by a test harness or reviewed by a reliability engineer before automation.

textCopy
You are a reliability test engineer designing retry behavior tests for an AI agent that calls external tools.

Your task is to generate a structured test plan that validates the agent's retry logic against the provided tool contract, failure modes, and retry policy.

## INPUTS

### Tool Contract
[TOOL_CONTRACT]

### Failure Modes to Test
[FAILURE_MODES]

### Retry Policy
[RETRY_POLICY]

### Idempotency Configuration
[IDEMPOTENCY_CONFIG]

### Additional Constraints
[CONSTRAINTS]

## OUTPUT SCHEMA

Return a JSON object with the following structure:

{
  "test_plan_id": "string",
  "tool_under_test": "string",
  "retry_policy_summary": "string",
  "scenarios": [
    {
      "scenario_id": "string",
      "failure_mode": "string",
      "failure_type": "transient | persistent | mixed",
      "injected_fault": {
        "attempt": "integer",
        "error_code": "string",
        "error_message": "string",
        "latency_ms": "integer",
        "response_body": "object | null"
      },
      "expected_agent_behavior": {
        "should_retry": "boolean",
        "max_retries_expected": "integer",
        "backoff_strategy": "string",
        "idempotency_key_behavior": "reuse | regenerate | none",
        "escalation_trigger": "string | null",
        "fallback_action": "string | null",
        "user_communication": "string | null"
      },
      "assertions": [
        {
          "assertion_id": "string",
          "condition": "string",
          "expected_value": "string | boolean | integer",
          "measurement_point": "string"
        }
      ],
      "pass_criteria": "string",
      "failure_impact": "string"
    }
  ],
  "idempotency_tests": [
    {
      "test_id": "string",
      "description": "string",
      "key_reuse_scenario": "string",
      "expected_deduplication": "boolean",
      "side_effect_check": "string"
    }
  ],
  "global_assertions": [
    "string"
  ],
  "test_execution_order": ["string"],
  "known_limitations": ["string"]
}

## INSTRUCTIONS

1. Parse the tool contract to identify all endpoints, methods, and expected response shapes.
2. For each failure mode in the input list, generate at least one transient scenario (resolves before max retries) and one persistent scenario (exceeds max retries).
3. Map each scenario to the retry policy: max attempts, backoff multiplier, jitter, and retryable error codes.
4. Include idempotency key reuse checks where the tool contract specifies idempotency support.
5. For persistent failures, define the escalation path and fallback behavior explicitly.
6. Write assertions that can be verified in logs or traces, not just output inspection.
7. Flag any gaps between the retry policy and the failure modes that need human review.

## RISK LEVEL: [RISK_LEVEL]

If RISK_LEVEL is "high", add a human review gate before any destructive or state-mutating retry scenario and include rollback verification assertions.

Adapt this template by replacing each square-bracket placeholder with your concrete specifications. The [TOOL_CONTRACT] should include the full API schema, authentication method, and expected response codes. The [FAILURE_MODES] should enumerate specific error conditions you want to test—timeouts, 429 rate limits, 503 service unavailable, malformed JSON responses, and authentication expiry. The [RETRY_POLICY] must specify max attempts, backoff strategy (exponential, linear, fixed), jitter configuration, and which error codes are retryable versus non-retryable. The [IDEMPOTENCY_CONFIG] should describe how idempotency keys are generated, stored, and validated. Set [RISK_LEVEL] to "high" for tools that mutate state, handle payments, or operate in regulated domains—this activates additional safety assertions and human-review gates in the generated test plan.

After generating the test plan, validate it against your actual retry implementation. Common gaps include: missing tests for idempotency key collisions, no scenarios for partial success responses, and retry policies that don't account for tool-side deduplication windows. Run the generated scenarios in a staging environment before promoting to your CI pipeline. If the tool contract changes, regenerate the test plan and diff it against the previous version to catch regressions in retry coverage.

This prompt is designed to be run as part of a pre-release test suite for any agent that interacts with external tools. The primary integration point is a test runner that can call an LLM, parse the generated test scenarios, and then execute those scenarios against the agent under test. Do not run this prompt in production; it is a development-time quality gate. The prompt expects a tool contract (OpenAPI, JSON Schema, or a structured description of the tool's behavior) and a retry policy document as inputs. The output is a machine-readable test plan that can be fed into a test executor.

To wire this into a CI pipeline, wrap the prompt in a script that: (1) loads the tool contract and retry policy from the repository, (2) substitutes them into the [TOOL_CONTRACT] and [RETRY_POLICY] placeholders, (3) calls a capable model (GPT-4o, Claude 3.5 Sonnet, or equivalent) with a low temperature (0.1–0.2) to maximize deterministic output, (4) parses the JSON output and validates it against the expected test scenario schema, and (5) writes the validated scenarios to a test artifacts directory. For manual QA, the same prompt can be used in a chat interface, but you must still validate the output schema before executing the tests. Add a retry wrapper around the LLM call itself: if the model returns malformed JSON, retry once with the error message appended to the prompt. If it fails again, fail the pipeline and alert the developer.

The generated test scenarios should be executed by a separate test harness that mocks the tool endpoint. Each scenario defines a failure injection (e.g., 'return 503 on attempt 1, success on attempt 2') and an expected agent behavior (e.g., 'retries once with exponential backoff, then returns result'). The test harness must verify that the agent's actual retry count, backoff timing, idempotency key reuse, and escalation behavior match the expected values. Log every tool call attempt with its timestamp, idempotency key, and response status. After the test run, compare the log against the expected behavior from the prompt output. Fail any scenario where the agent retried too many times, too few times, reused an idempotency key incorrectly, or escalated prematurely. For high-risk tools (write operations, financial transactions), add a human review step before promoting the agent to production, even if all automated tests pass.