Agent Tool Idempotency Test Prompt

This prompt is for integration developers and AI quality engineers who need to verify that an agent's tool calls are safe to retry. When an agent invokes a payment API, a database write, or a resource provisioning endpoint, a network timeout can leave the system in an unknown state. Retrying without idempotency guarantees can double-charge a customer, create duplicate records, or corrupt state. This prompt generates a structured test plan that invokes the same tool call multiple times with identical idempotency keys and validates that the outcome is consistent, side-effect-free, and matches the declared idempotency contract. Use this prompt before deploying any agent that calls mutating external tools. It belongs in your pre-release test suite, not as a runtime guard.

The ideal user has access to the tool's API contract, including the idempotency key field name, the HTTP method, and the expected success and conflict responses. You will need to provide the tool's OpenAPI spec or equivalent schema, the idempotency key parameter, and the target environment (staging only—never run idempotency tests against production payment or provisioning endpoints). The prompt expects a [TOOL_CONTRACT] that defines the endpoint, the [IDEMPOTENCY_KEY_FIELD], and the [TEST_ENVIRONMENT] where the tests will execute. Without a clear idempotency contract from the tool provider, this prompt cannot produce valid test cases; in that scenario, first use the Tool Contract Validation Prompt to verify the contract exists.

Do not use this prompt for read-only GET operations, for tools that lack an explicit idempotency mechanism, or for workflows where the agent's retry logic is handled entirely by a message queue with exactly-once delivery guarantees. This prompt is also inappropriate for runtime use—it generates test plans, not live guardrails. If you need runtime idempotency enforcement, implement that in your API gateway or tool proxy layer, not in the agent's prompt. After generating the test plan, execute it in a sandboxed staging environment, review the results for any duplicate side effects, and ensure your agent's retry logic respects the idempotency keys before promoting to production.

This prompt template generates a structured test plan that invokes the same tool call multiple times with identical idempotency keys and validates that the outcome is consistent, side-effect-free, and matches the declared idempotency contract. Use it when you need to verify that a tool's retry behavior is safe before deploying an agent that may retry failed or timed-out calls. The template expects a tool contract as input and produces a set of test scenarios, each with a sequence of identical calls, expected responses, and pass/fail assertions.

textCopy
You are an idempotency test designer for agent tool integrations. Your job is to generate a test plan that verifies a tool's idempotency behavior under repeated identical invocations.

## INPUT

[TOOL_CONTRACT]

## CONSTRAINTS

- The tool contract includes: name, description, argument schema, output schema, and a declared idempotency policy (key field, key source, retry window, expected behavior on duplicate calls).
- Generate test cases that exercise the idempotency contract, not generic retry behavior.
- Each test case must specify the exact arguments, the idempotency key value, the number of repeated calls, and the expected outcome for each call.
- Cover these scenarios: identical key within retry window, identical key outside retry window, missing key, malformed key, key reuse across different argument sets, and concurrent duplicate calls.
- For each scenario, define whether the expected behavior is: same result returned, operation executed once, error returned, or behavior undefined.
- If the tool contract does not declare an idempotency policy, flag it as a contract gap and generate test cases that detect accidental non-idempotent behavior.
- Do not generate destructive test cases that mutate production data. Mark any test that requires a live system with [REQUIRES_ISOLATED_ENVIRONMENT].

## OUTPUT SCHEMA

Return a JSON object with this structure:
{
  "tool_name": "string",
  "idempotency_policy_summary": "string describing the declared policy or noting its absence",
  "contract_gaps": ["string flags for missing or ambiguous policy elements"],
  "test_cases": [
    {
      "test_id": "string",
      "scenario": "string describing the idempotency scenario being tested",
      "idempotency_key": "string or null",
      "arguments": {},
      "call_count": number,
      "expected_outcomes": [
        {
          "call_index": number,
          "expected_http_status": number,
          "expected_response_signature": "string describing key fields that must match",
          "expected_side_effects": "string describing allowed or disallowed mutations",
          "idempotency_behavior": "one of: first_execution | duplicate_acknowledged | error | undefined"
        }
      ],
      "assertions": ["string pass/fail conditions"],
      "requires_isolated_environment": boolean
    }
  ],
  "eval_criteria": {
    "pass_conditions": ["string conditions that must all be true for the tool to pass"],
    "fail_conditions": ["string conditions that indicate a violation of the idempotency contract"],
    "warning_conditions": ["string conditions that are acceptable but worth monitoring"]
  }
}

## EXAMPLES

[EXAMPLES]

## RISK LEVEL

[RISK_LEVEL]

Generate the test plan now.

Adapt this template by replacing [TOOL_CONTRACT] with the full tool definition—including its argument schema, output schema, and any documented idempotency behavior. Populate [EXAMPLES] with one or two representative test cases if you have known patterns the model should follow. Set [RISK_LEVEL] to low, medium, or high to control the conservatism of the generated assertions; high-risk tools will get stricter pass/fail criteria and more explicit warnings about side-effect detection. After generation, validate the output against the declared JSON schema before feeding test cases into your test harness. For tools that mutate external state, always run generated tests in an isolated environment first.

The idempotency test prompt is designed to be invoked programmatically as part of a tool contract verification suite. It expects a structured input containing the tool's contract (description, argument schema, idempotency key field), the tool's declared idempotency behavior, and any relevant constraints such as rate limits or side-effect warnings. The output is a set of test cases—each specifying the tool call arguments, the idempotency key value, the number of repeated invocations, and the expected outcome—that can be fed into a test runner or manual review queue. This prompt is not a one-shot chat interaction; it is a generation step in a larger testing pipeline.

To integrate this prompt into an application, wrap it in a function that accepts the tool contract and configuration as input, calls the model with a low temperature (0.0–0.2) to maximize deterministic output, and parses the response into a typed array of test case objects. Each test case must include a unique test_id, the tool_name, arguments (with the idempotency key field populated), repeat_count, and expected_outcome (e.g., 'identical_result', 'no_duplicate_side_effect', 'idempotency_key_rejected_if_missing'). Validate the output against this schema immediately after generation. If validation fails, retry once with the validation errors appended to the prompt as [CONSTRAINTS]. Log every generation attempt, the raw model output, the parsed result, and any validation errors to your observability stack for debugging and audit. For high-risk tools (e.g., payment APIs, database writes, provisioning systems), route the generated test cases through a human approval step before execution. The approval UI should display the tool contract, the generated test cases, and a diff against any previously approved test set.

When executing the generated test cases, use a dedicated test harness that can invoke the actual tool or a mock with identical idempotency behavior. The harness must track the idempotency key across repeated calls and assert that the outcome matches the expected behavior defined in the test case. Common failure modes to monitor include: the model generating test cases that don't exercise the idempotency key field at all, producing duplicate test cases with different test_id values, or hallucinating expected outcomes that contradict the tool's documented contract. To catch these, add eval checks that verify every generated test case references the declared idempotency key field, that the set of test cases is deduplicated by argument signature, and that each expected outcome is consistent with the tool's stated idempotency guarantee. If the tool contract changes, re-run the prompt and diff the new test cases against the previous set to detect regressions or gaps in coverage.