Agent Confidence Score Annotation Prompt

This prompt is designed for evaluation engineers and platform builders who need to standardize how agents communicate uncertainty. The core job-to-be-done is transforming a raw agent output and its internal reasoning trace into a structured, machine-readable confidence payload that includes a calibrated score, evidence strength indicators, and threshold-based action triggers. You should use this prompt when you are building a multi-agent system where downstream agents or orchestration layers must make automated routing, escalation, or fusion decisions based on the reliability of an upstream agent's work. It is not a general-purpose 'how confident are you' question for a chatbot; it is a strict annotation contract for production pipelines.

Required context for this prompt to work reliably includes: the agent's final output, the evidence or sources the agent used, a defined confidence scale (e.g., 0.0–1.0 or Low/Medium/High), and pre-defined action thresholds that map score ranges to behaviors like 'auto-approve,' 'escalate to human,' or 'route to a secondary agent for verification.' Without these inputs, the model will invent its own calibration criteria, leading to inconsistent scores across agents. Do not use this prompt for real-time, latency-sensitive chat where a structured annotation step would degrade user experience, or in isolation without a calibration eval loop that compares annotated scores against ground-truth outcomes over time.

The primary risk is false confidence: an agent may produce a high-confidence score for a fluent but factually incorrect output. To mitigate this, the prompt forces the model to separate the confidence score from the evidence strength rating and to cite specific gaps or ambiguities. After implementing this prompt, your next step should be to log every annotation alongside the eventual human judgment or business outcome, then run a calibration evaluation (e.g., Expected Calibration Error) to detect overconfidence drift. Avoid treating the annotated score as a static quality gate without this ongoing measurement.

This prompt template is designed to be inserted into an agent's response generation step, forcing it to annotate its own output with a machine-readable confidence payload. The primary job is to standardize uncertainty communication so that downstream agents, orchestrators, or human reviewers can make informed decisions about whether to trust, escalate, or discard the agent's work. Use this template when you need every agent in a multi-agent system to speak the same 'uncertainty language'—especially before a handoff or an action with side effects.

textCopy
You are an agent operating within a multi-agent system. Your task is to produce a final answer for the user, but you must also annotate that answer with a structured confidence assessment. This annotation will be parsed by downstream systems to decide whether to accept your output, escalate it, or request a retry.

First, generate your answer to the user's request based on the provided context.

Then, you MUST append a single, valid JSON object wrapped in <CONFIDENCE> tags. Do not include any other text inside the tags. The JSON object must conform to this schema:

{
  "confidence_score": <float 0.0 to 1.0>,
  "calibration_notes": "<string: explain your certainty level, e.g., 'High confidence: all facts are directly cited from the provided document.' or 'Low confidence: the user's request is ambiguous and I am inferring intent.'>",
  "evidence_strength": "<enum: STRONG | PARTIAL | WEAK | NONE>",
  "evidence_summary": "<string: a concise summary of the key evidence supporting your answer>",
  "missing_information": ["<string: list any critical information that was not available but would have improved the answer>"],
  "recommended_action": "<enum: ACCEPT | REVIEW | ESCALATE | RETRY>"
}

[CONSTRAINTS]
- The confidence_score must reflect your internal, calibrated certainty. Do not default to 1.0.
- If evidence_strength is WEAK or NONE, the recommended_action must not be ACCEPT.
- If you are unsure or the request is ambiguous, set recommended_action to REVIEW or ESCALATE and explain why in calibration_notes.
- The answer and the <CONFIDENCE> block must be the only output.

[INPUT]
User Request: [USER_REQUEST]
Available Context: [CONTEXT]

[EXAMPLES]
User Request: What is the capital of France?
Available Context: Paris is the capital and most populous city of France.
Answer: The capital of France is Paris.
<CONFIDENCE>
{
  "confidence_score": 0.99,
  "calibration_notes": "Directly stated in the provided context.",
  "evidence_strength": "STRONG",
  "evidence_summary": "The context explicitly states 'Paris is the capital... of France.'",
  "missing_information": [],
  "recommended_action": "ACCEPT"
}
</CONFIDENCE>

User Request: What is the best restaurant in Tokyo?
Available Context: Tokyo has many highly-rated restaurants. Sushi Saito is a famous three-Michelin-starred sushi restaurant.
Answer: While 'best' is subjective, Sushi Saito is a highly acclaimed three-Michelin-starred sushi restaurant in Tokyo.
<CONFIDENCE>
{
  "confidence_score": 0.4,
  "calibration_notes": "The question is subjective. I can provide a famous example but cannot determine the objective 'best'.",
  "evidence_strength": "PARTIAL",
  "evidence_summary": "Context mentions Sushi Saito as a famous restaurant but provides no comparative data.",
  "missing_information": ["User's cuisine preference", "User's budget", "Comparative reviews"],
  "recommended_action": "REVIEW"
}
</CONFIDENCE>

To adapt this template, start by tuning the recommended_action enum values to match your system's specific routing and escalation logic. You may need to add a risk_level field if the agent operates in a regulated domain. The most critical part to test is the model's calibration: run a batch of known-answer and ambiguous queries through this prompt and plot the confidence_score against actual accuracy to detect overconfidence. If the model consistently assigns high scores to wrong answers, add stronger language in the [CONSTRAINTS] section or provide more few-shot examples of low-confidence, correct behavior. For high-risk workflows, always route outputs with recommended_action: REVIEW or ESCALATE to a human queue before any destructive action is taken.

This prompt is designed to be called programmatically as part of an automated evaluation harness, not as a one-off chat interaction. The primary integration point is immediately after an agent produces a final answer or a critical intermediate claim. The application layer should extract the agent's output, wrap it in the [AGENT_OUTPUT] placeholder along with the [TASK_CONTEXT] and [EVIDENCE_SNIPPETS], and call the LLM to produce the structured confidence annotation. The resulting JSON payload should be parsed and stored alongside the agent's output in your tracing or evaluation database. Do not rely on the agent to self-report confidence inline; use this prompt as a separate, independent evaluation step to avoid self-assessment bias.

For production wiring, implement a retry wrapper around the LLM call that validates the output against the expected JSON schema before accepting it. If the model returns malformed JSON, missing required fields like confidence_score or evidence_strength, or values outside the 0.0–1.0 range, discard the response and retry with a stricter prompt variant that includes the validation error in the next request. Log every annotation attempt—including raw LLM responses, parse errors, and retry counts—to your observability platform. This log becomes critical for debugging calibration drift over time. For high-stakes domains such as healthcare or finance, route annotations with a confidence_score below a configurable threshold (e.g., 0.7) or a recommended_action of ESCALATE to a human review queue before the agent's output is surfaced to users or written to a system of record.

Model choice matters here. Use a model with strong instruction-following and JSON mode support (e.g., GPT-4o, Claude 3.5 Sonnet) for the annotation step, even if the underlying agent uses a smaller or faster model. The annotation model should be treated as an evaluator, and its quality should be measured independently using a golden dataset of human-annotated confidence examples. Run periodic calibration checks by comparing the model's confidence_score against actual outcome data (e.g., was the agent's claim factually correct?). If you observe systematic overconfidence, adjust the prompt's calibration instructions or add few-shot examples that demonstrate appropriate uncertainty. Avoid wiring this prompt directly into a user-facing synchronous loop without caching or batching, as the additional LLM call adds latency and cost to every agent turn.