Error Handling and Retry Logic Evaluation Prompt

This prompt is built for evaluation engineers and agent platform teams who need to measure how an agent behaves when external tool calls fail. It is not a prompt you ship to end users; it is a prompt you run against agent traces in a test harness. The job-to-be-done is automated, repeatable grading of error-handling quality before you deploy an agent update. You should use this prompt when you have a recorded trace of an agent interacting with tools, that trace contains at least one failed tool call (timeout, permission denied, malformed response, or rate limit), and you need a structured score for error detection, retry decision quality, backoff behavior, and escalation correctness. The ideal user is someone who owns agent reliability and wants to replace manual spot-checking of error traces with a consistent, model-graded evaluation that can run in CI/CD or as part of a regression suite.

Do not use this prompt when you need to evaluate tool selection correctness, argument accuracy, or call sequencing in successful tool calls. Those concerns belong to separate evaluation surfaces such as the Tool Selection Correctness Grading Prompt or the Function Argument Accuracy Evaluation Rubric. This prompt also should not be used to evaluate the agent's final answer quality or task completion rate; it focuses narrowly on the error-recovery path. If your trace contains only successful tool calls, this prompt will have nothing to grade. If your agent does not have retry logic, backoff configuration, or escalation paths, the prompt will still produce scores, but they will reflect the absence of those capabilities rather than their quality. In high-risk domains where error-handling failures could cause financial loss, data corruption, or safety incidents, pair this prompt's output with human review of a sample of graded traces and track score distributions over time to detect regressions.

Before using this prompt, ensure you have a representative set of agent traces that include diverse failure modes. A trace that only contains timeouts will not tell you how the agent handles permission errors. Build your evaluation dataset to cover the failure categories your agent is expected to encounter in production. Wire this prompt into a harness that extracts the relevant trace segments, formats them into the [AGENT_TRACE] placeholder, and collects the structured scores for aggregation. The next section provides the copy-ready prompt template you can adapt for your specific agent architecture and failure taxonomy.

This prompt template evaluates how an AI agent handles tool-call failures. It scores the agent's ability to detect errors, decide whether to retry, apply appropriate backoff, and escalate when retries are exhausted. The template is designed to be dropped into an evaluation harness where you supply the agent's tool-call trace, the failure scenario, and your expected behavior criteria. Use it to catch brittle retry loops, missing error detection, and premature escalation before these failures reach production.

textCopy
You are an evaluation judge grading an AI agent's error handling and retry behavior during a tool-use interaction.

## Evaluation Context
[FAILURE_SCENARIO]: Description of the simulated failure (e.g., timeout after 30s, HTTP 403, malformed JSON response, rate limit 429).
[AGENT_TRACE]: The full agent trace including the initial tool call, any error response, subsequent retry attempts, and final action.
[TOOL_CATALOG]: The list of available tools with their schemas and documented error modes.
[RETRY_POLICY]: The expected retry policy (e.g., max 3 retries, exponential backoff starting at 1s, escalate after exhaustion).
[ESCALATION_PATH]: The expected escalation behavior when retries are exhausted (e.g., return error to user, call fallback tool, log and abort).

## Scoring Rubric
Grade the agent on a 1-5 scale for each dimension:

### 1. Error Detection (1-5)
- 5: Agent correctly identified the specific error type and extracted relevant details (status code, error message, retry-after header).
- 3: Agent detected that a failure occurred but misclassified the error type or missed important details.
- 1: Agent did not detect the failure and proceeded as if the call succeeded.

### 2. Retry Decision (1-5)
- 5: Agent correctly decided whether to retry based on error type (retried on transient errors like 429/503, did not retry on permanent errors like 400/403).
- 3: Agent retried appropriately for some errors but missed the distinction for others.
- 1: Agent retried on permanent errors or failed to retry on clearly transient errors.

### 3. Backoff Behavior (1-5)
- 5: Agent applied appropriate backoff strategy matching the retry policy (exponential, respecting retry-after headers, jitter).
- 3: Agent retried with some delay but did not follow the specified backoff strategy.
- 1: Agent retried immediately with no backoff or used fixed intervals contrary to policy.

### 4. Escalation Correctness (1-5)
- 5: Agent escalated correctly after exhausting retries, following the specified escalation path with complete context.
- 3: Agent escalated but with incomplete context or after the wrong number of retries.
- 1: Agent did not escalate when required or escalated prematurely before retries were exhausted.

### 5. Context Preservation (1-5)
- 5: Agent preserved all relevant context through retries and escalation (original intent, partial results, error history).
- 3: Agent preserved some context but lost important details across retry attempts.
- 1: Agent lost critical context, making escalation or debugging impossible.

## Output Format
Return a JSON object with this exact schema:
{
  "error_detection": { "score": <1-5>, "justification": "<string>" },
  "retry_decision": { "score": <1-5>, "justification": "<string>" },
  "backoff_behavior": { "score": <1-5>, "justification": "<string>" },
  "escalation_correctness": { "score": <1-5>, "justification": "<string>" },
  "context_preservation": { "score": <1-5>, "justification": "<string>" },
  "aggregate_score": <float 1.0-5.0>,
  "failure_mode_summary": "<string describing the primary failure pattern, or 'none' if all scores >= 4>",
  "recommendation": "<string: 'pass', 'review', or 'fail'>"
}

## Constraints
- Score only the agent's behavior against the provided retry policy and escalation path.
- Do not penalize the agent for tool-call failures caused by the simulated scenario.
- If the agent trace is incomplete or ambiguous, note this in your justifications and score conservatively.
- The aggregate_score is the mean of the five dimension scores.
- Recommend 'pass' if aggregate >= 4.0, 'review' if >= 3.0, 'fail' if < 3.0.

To adapt this template, replace each square-bracket placeholder with your test scenario data. The [FAILURE_SCENARIO] should describe exactly what went wrong—be specific about error codes, timing, and response payloads. The [AGENT_TRACE] is the raw output from your agent framework showing every step. If your retry policy differs from the rubric's assumptions (e.g., you allow more retries or use circuit breakers), adjust the scoring descriptions accordingly. For high-stakes production systems, always pair this prompt with human review of failing traces and calibrate the judge against manual annotations before relying on automated pass/fail decisions.

This evaluation prompt is designed to run as a post-execution judge inside an agent trace analysis pipeline. After an agent completes a multi-step tool-use task—or fails partway through—you feed the full trace (tool calls, tool responses, agent reasoning, and final output) into this prompt along with a structured test scenario definition. The prompt returns a scored evaluation across four dimensions: error detection, retry decision appropriateness, backoff behavior, and escalation correctness. Wire this as a batch evaluation step that runs after each test suite execution, not as a real-time guard during live agent runs. The latency and cost profile of a detailed LLM judge is better suited for offline analysis and regression testing than for inline gating.

Integration pattern: Store test scenarios as structured JSON objects, each containing a scenario_id, error_type (e.g., timeout, permission_denied, malformed_response), expected_behavior (e.g., retry_with_backoff, escalate, abort), and the raw agent trace. Feed each scenario through the evaluation prompt and collect the four sub-scores plus the overall error_handling_score. Validate the output against a strict JSON schema before accepting it—reject and retry the judge call if fields are missing, types are wrong, or scores fall outside the 0.0–1.0 range. Log every evaluation result with the scenario_id, model_version, prompt_version, and timestamp so you can track score drift across prompt changes and model upgrades.

Retry and fallback for the judge itself: The evaluation prompt can fail—malformed JSON, refusal on sensitive-looking traces, or timeouts on very long agent transcripts. Implement a lightweight retry wrapper: up to 2 retries with exponential backoff (1s, 4s) if the output fails schema validation or the model returns an error. If the judge still fails after retries, flag the scenario for human review rather than silently assigning a default score. This is especially important for permission_denied and malformed_response scenarios where the agent's behavior may have security or data integrity implications. Store failed evaluations in a separate review queue with the raw trace and the judge's partial output attached.

Model choice and cost control: Use a capable model (GPT-4o, Claude 3.5 Sonnet, or equivalent) for the judge because error-handling evaluation requires nuanced reasoning about whether a retry was justified, whether backoff timing was appropriate, and whether escalation matched policy. Do not use a smaller or faster model for this judge unless you've calibrated its scores against human ratings and confirmed acceptable agreement. To manage costs, sample traces rather than evaluating every single agent run—evaluate 100% of runs during regression testing and prompt development, then drop to a statistically meaningful sample in production monitoring. Cache evaluation results by scenario_id + trace_hash to avoid re-evaluating identical traces.

Human review integration: For any scenario where the error_handling_score falls below 0.6 or where the escalation_correctness sub-score is below 0.5, route the evaluation to a human reviewer before accepting the result into your regression baseline. These thresholds indicate the agent either mishandled the error or the judge is uncertain—both cases warrant human inspection. Build a simple review UI that shows the agent trace, the judge's scores, and the judge's justification side by side. Let reviewers override scores with a note, and feed those overrides back into your calibration dataset to improve the judge prompt over time.

What to avoid: Do not use this evaluation prompt as a real-time circuit breaker during live agent execution—the latency is too high and the judge's own failure modes would compound the agent's error. Do not treat the judge's scores as ground truth without periodic calibration against human ratings, especially when you change the underlying model. Do not skip schema validation on the judge's output; an unvalidated score that slips into a dashboard or regression report will erode trust in your entire evaluation pipeline. Start with a small set of 10–15 hand-annotated test scenarios, calibrate the judge against them, and expand the scenario library as you discover new failure modes in production traces.