Uncertainty Metadata Annotation Prompt for API Responses

This prompt is designed for API product teams that need to expose machine-readable uncertainty signals alongside AI-generated responses. Instead of returning only a text answer, the system annotates the payload with structured metadata: a confidence score, an evidence coverage ratio, and explicit gap flags. Downstream services, monitoring dashboards, and routing logic consume these signals to decide whether to auto-respond, queue for human review, or refuse. Use this prompt when your API contract requires uncertainty metadata that other machines can parse and act on.

The ideal integration point is a post-generation annotation layer within a RAG pipeline or any AI service where the raw text response is insufficient for operational decision-making. You should wire this prompt to run after answer generation but before the final API response is serialized. The required inputs are the generated answer text and the set of retrieved evidence passages used to produce it. The output is a strictly typed JSON object containing a confidence_score (0.0–1.0), an evidence_coverage_ratio (0.0–1.0), a list of gap_flags for unsupported claims, and a sufficiency_tier enum (e.g., HIGH, MEDIUM, LOW, INSUFFICIENT). This structured output allows a routing middleware to implement rules like: if sufficiency_tier is LOW or INSUFFICIENT, escalate to a human review queue and log the gap flags for later diagnosis.

Do not use this prompt for direct user-facing chat where raw scores would confuse non-technical users; pair it with a separate formatting layer if the same response must serve both audiences. Avoid applying it to purely creative or open-ended generation tasks where 'evidence' is undefined. The prompt is also not a substitute for upstream retrieval quality—if your retriever consistently returns irrelevant passages, the annotation will correctly flag low coverage, but the root cause must be fixed in retrieval, not in the metadata layer. For high-stakes domains such as healthcare or legal, always require human review of any response flagged below HIGH sufficiency, and log the full annotation payload for audit trails.

The following prompt template is designed to be dropped directly into an API workflow. It instructs the model to act as an uncertainty annotation engine, analyzing a generated answer against a provided evidence list and returning a strict JSON payload. The prompt does not rewrite the answer or produce conversational text; its sole job is to produce machine-readable metadata for downstream consumers like logging systems, user-facing confidence displays, or automated release gates.

textCopy
You are an uncertainty annotation engine for an API. Your task is to analyze a generated answer against its source evidence and produce a structured uncertainty metadata payload. You must not rewrite the answer. You must not produce natural-language commentary outside the metadata fields.

INPUTS:
- [QUERY]: the original user question
- [ANSWER]: the generated answer text to evaluate
- [EVIDENCE_LIST]: an array of evidence objects, each with `id`, `text`, `relevance_score`, and `source_authority`

INSTRUCTIONS:
1. For each claim in [ANSWER], determine whether it is directly supported by at least one passage in [EVIDENCE_LIST]. A claim is supported if the evidence text contains the same factual statement, not merely a related topic.
2. Count the total number of claims in [ANSWER] and the number of supported claims.
3. Compute `evidence_coverage_ratio` as `supported_claims / total_claims`. If total_claims is 0, set coverage to 1.0.
4. Assign a `confidence_score` between 0.0 and 1.0 based on: coverage ratio, average relevance_score of supporting evidence, and source_authority of supporting evidence. Use this heuristic: start at coverage ratio, then reduce by 0.1 if average relevance_score is below 0.7, and reduce by 0.1 if no supporting source has `source_authority` of 'high'.
5. Identify `gap_flags`: for each unsupported claim, create a flag with `claim_text`, `reason` (one of 'no_evidence', 'contradicted', 'speculative'), and `severity` ('critical', 'major', 'minor').
6. Set `overall_assessment` to 'grounded' if confidence_score >= 0.8, 'partial' if >= 0.5, 'insufficient' otherwise.

OUTPUT SCHEMA:
Return ONLY a valid JSON object with this exact structure:
{
  "uncertainty_metadata": {
    "confidence_score": float,
    "evidence_coverage_ratio": float,
    "total_claims": integer,
    "supported_claims": integer,
    "gap_flags": [
      {
        "claim_text": string,
        "reason": "no_evidence" | "contradicted" | "speculative",
        "severity": "critical" | "major" | "minor"
      }
    ],
    "overall_assessment": "grounded" | "partial" | "insufficient"
  }
}

Do not include any text outside the JSON object.

To adapt this template for your own system, replace the placeholders with your actual data at runtime. The [QUERY] and [ANSWER] fields are straightforward string substitutions. The [EVIDENCE_LIST] must be serialized as a JSON array of objects matching the specified schema. If your retrieval pipeline uses different field names (e.g., score instead of relevance_score), map them before injection or adjust the prompt's field references accordingly. The confidence heuristic can be tuned by modifying the penalty thresholds (0.7 relevance, 'high' authority) to match your domain's evidence quality distribution. For high-stakes domains, consider adding a [RISK_LEVEL] parameter that adjusts the overall_assessment thresholds upward, requiring higher confidence for 'grounded' classification.

Before deploying, validate that your model reliably produces parseable JSON matching the output schema. Common failure modes include the model wrapping the JSON in markdown fences, adding explanatory text before or after the object, or hallucinating additional fields. Implement a post-processing validator that strips markdown fences, parses the JSON, and checks for schema compliance. If the output fails validation, retry with a stricter instruction appended to the prompt, such as 'Your previous response was not valid JSON. Return ONLY the JSON object with no other text.' After three failed retries, log the failure and escalate for human review rather than silently passing malformed metadata to downstream consumers.

Integrate the Uncertainty Metadata Annotation Prompt as a post-generation annotation step in your RAG pipeline. After the primary model produces [ANSWER], invoke this prompt with the original [QUERY], the generated [ANSWER], and the [EVIDENCE_LIST] used during generation. This step produces structured uncertainty metadata—confidence scores, evidence coverage ratios, and gap flags—that downstream systems can consume programmatically. The annotation step should be treated as a separate model call with its own latency budget, error handling, and observability, not as an inline addition to the primary generation prompt.

Use structured output mode (JSON mode) if your model provider supports it. Configure the API call to enforce the output schema defined in the prompt template, which includes fields like overall_assessment, confidence_score, evidence_coverage_ratio, and gap_flags. If the model does not support native structured output, parse the raw response and validate it against the expected schema. On parse failure, retry once with the original prompt plus the validation error message appended as additional context. Log every annotation result—including the raw response, parsed output, validation status, and retry count—alongside the request ID for traceability. If overall_assessment equals 'insufficient', route the response to a human review queue or trigger a refusal workflow rather than serving the answer to the end user.

Implement caching to avoid redundant computation. Cache annotation results keyed by a composite hash of (query_hash, answer_hash, evidence_hash). This is especially valuable when identical or near-identical requests pass through the pipeline repeatedly. Set a latency budget of 500ms for the annotation step. If the call exceeds this threshold, fall back to a default 'partial' assessment with a warning flag logged for monitoring. This prevents the annotation step from becoming a bottleneck while still surfacing degraded annotation quality to your observability stack. Monitor the rate of timeout fallbacks, parse failures, and 'insufficient' assessments as key health metrics for your grounding pipeline.