Tool Output Confidence Scoring Prompt After Error

This prompt is designed for agent developers and reliability engineers who are building production tool-calling loops where failures are a routine operational reality. The core job-to-be-done is making a safe, structured decision about whether to trust recovered or partial tool output before it contaminates downstream systems. After a tool call fails and you have recovered data through retries, fallback mechanisms, or partial result extraction, you cannot simply pass that data forward blindly. This prompt evaluates the error context, the recovery method used, and the completeness of the data to produce a structured confidence score and a binary safe_for_downstream decision. The ideal user is an AI platform engineer who understands that the cost of acting on bad data—such as corrupting a database, sending a faulty command, or presenting false information to a user—far exceeds the cost of discarding potentially recoverable data.

To use this prompt effectively, you must provide it with specific, structured inputs. These include the raw error output, the recovery method employed (e.g., retry_with_modified_args, fallback_to_cached_result, partial_parse), the recovered data payload, and the expected output schema. The prompt is designed to be wired into an automated harness where it acts as a gating function. For example, in an agent loop, the output of a tool call is caught by an error handler, which then attempts a recovery strategy. Before the recovered data is passed to the next tool or used to formulate a user response, this prompt is called. Its safe_for_downstream boolean output directly controls a conditional gate in the application code, preventing unsafe data from proceeding. This is not a prompt for real-time, safety-critical systems where any uncertainty requires immediate human review; in those contexts, a safe_for_downstream value of false should trigger an immediate escalation to a human operator, not another automated fallback.

You should not use this prompt when the failure mode is unknown or the error context is empty. The prompt's evaluation logic depends on understanding how the data was recovered to assess its reliability; a partial_parse of a malformed JSON response carries a different risk profile than a successful retry with modified arguments. Avoid using this prompt as a substitute for proper output validation. Schema validation should always occur before this confidence scoring step. The prompt's job is to assess the semantic trustworthiness of already-validated data, not to check for structural correctness. Finally, do not use this prompt for non-deterministic or creative generation tasks where there is no ground truth for 'correct' output. Its value is in high-stakes, structured data pipelines where a single bad record can have cascading effects. The next step after implementing this prompt is to pair it with an evaluation harness that specifically tests for overconfidence after recovery and unnecessary rejection of good data.

This prompt template is designed to be pasted directly into your agent's evaluation step immediately after a tool call has failed and been recovered, or has returned partial data. Its job is to produce a structured confidence score that your application can use to decide whether the output is safe for downstream consumption. The prompt forces the model to reason about the error context, the recovery method, and the completeness of the data before assigning a score, preventing the common failure mode of blindly trusting a successful retry.

textCopy
You are an output integrity evaluator for an AI agent. Your task is to assess the confidence we should have in a tool's output after an error occurred.

**Tool Name:** [TOOL_NAME]
**Tool Description:** [TOOL_DESCRIPTION]
**Intended Use of Output:** [DOWNSTREAM_USE]

**Error Context:**
- Error Type: [ERROR_TYPE]
- Error Message: [ERROR_MESSAGE]
- Recovery Method: [RECOVERY_METHOD]
- Retry Count: [RETRY_COUNT]

**Output to Evaluate:**
```json
[TOOL_OUTPUT]

Expected Output Schema:

jsonCopy
[OUTPUT_SCHEMA]

Evaluation Criteria:

1. Schema Compliance: Does the output match the expected schema? Are all required fields present and of the correct type?
2. Data Completeness: Are there any null, empty, or truncated fields that should contain data? Is the volume of data consistent with a successful call?
3. Error Residue: Does the output contain any error messages, stack traces, or partial failure indicators?
4. Semantic Coherence: Does the data make logical sense given the tool's purpose? Are there any impossible values or contradictions?
5. Recovery Risk: Given the recovery method used, what is the risk that the data is stale, corrupted, or duplicated?

Instructions:

1. Analyze the output against each criterion.
2. Assign a confidence score between 0.0 and 1.0, where 1.0 is perfect confidence.
3. Provide a concise reason for the score, citing specific evidence from the output.
4. Flag any specific fields that are untrustworthy.

Output Format: Respond ONLY with a valid JSON object matching this schema: { "confidence_score": <float 0.0-1.0>, "score_reason": "<string>", "untrustworthy_fields": ["<field_path>", ...], "recommendation": "<ACCEPT | REJECT | HUMAN_REVIEW>" }

To adapt this template, replace the square-bracket placeholders at runtime. [TOOL_OUTPUT] should be the raw, serialized output from the tool after recovery. [OUTPUT_SCHEMA] should be the expected JSON Schema or a plain-text description of required fields. [RECOVERY_METHOD] is critical context—be specific (e.g., "retry with exponential backoff after 429," "used cached result from T-5min," "parsed partial JSON from truncated stream"). The [DOWNSTREAM_USE] field helps the model calibrate risk; a score of 0.8 might be acceptable for generating a draft summary but unacceptable for a financial transaction. For high-risk domains, always set a HUMAN_REVIEW threshold in your application logic that overrides the model's recommendation if the confidence score falls below it.

After pasting this prompt, you must validate the model's output against the declared JSON schema before allowing any downstream action. A common failure mode is the model returning a confidence score above your threshold but with an empty score_reason or an untrustworthy_fields list that contradicts the score. Implement a post-processing check: if the recommendation is ACCEPT but untrustworthy_fields is not empty, force a HUMAN_REVIEW outcome. Wire this prompt into a dedicated evaluation step that runs after every tool error recovery, and log the full evaluation payload for auditability.

This prompt is designed to sit directly after a tool error recovery attempt, receiving the raw or partial output alongside structured error context. It is not a standalone prompt but a decision node in a larger agent orchestration loop. The primary integration point is inside a tool execution wrapper that catches exceptions, applies a recovery strategy (retry, fallback, partial result extraction), and then calls this prompt to score the resulting output before passing it downstream. The agent should treat the confidence score as a gate: outputs scoring below a configurable threshold (e.g., 0.7) are blocked from automatic consumption and must follow an escalation path.

To wire this into an application, build a ToolOutputEvaluator class or function that accepts the raw tool output, the error context object (including exception type, recovery method, and retry count), and the expected output schema. The function constructs the prompt by injecting these values into the [OUTPUT_PAYLOAD], [ERROR_CONTEXT], and [EXPECTED_SCHEMA] placeholders. After receiving the model's structured response—which must include confidence_score, completeness_flags, and rationale—validate the JSON strictly. If the score is below your threshold, route the output to a human review queue or a secondary fallback tool. If the score is above the threshold but completeness_flags indicate missing required fields, treat it as a degraded result and attach a warning to the downstream context. Log every evaluation, including the score, rationale, and final routing decision, to create an audit trail for debugging confidence drift over time.

Model choice matters here. Use a model with strong instruction-following and structured output capabilities (e.g., GPT-4o, Claude 3.5 Sonnet) because the prompt requires nuanced reasoning about data integrity and error impact. Avoid smaller or older models that may default to overconfidence after a successful retry or underconfidence after a fallback. Implement a retry budget for the scoring prompt itself: if the model fails to return valid JSON after two attempts, default to a conservative score of 0.0 and escalate. For high-risk domains like finance or healthcare, never allow an automatic pass based solely on the model's confidence score; always require a secondary validation step, such as schema completeness checks and, where feasible, human sampling of scored outputs. The most common production failure mode is the model assigning high confidence to a well-formed but semantically wrong output after a fallback tool substituted different data—test for this explicitly in your evals.