Verification Confidence Report Generation Prompt

This prompt is for verification system architects who need to generate a structured, audit-ready confidence report for a batch of claims that have already been matched against evidence. It consumes claim-level verification results—including verdicts, evidence counts, source authority scores, and contradiction flags—and produces a report with calibrated confidence levels, factor breakdowns, and calibration notes. The primary job-to-be-done is translating raw verification signals into explainable confidence scores that downstream systems can use for triage, human review routing, or compliance documentation. Use this when you need to explain why a claim received a particular confidence level, not just a binary true/false label.

This prompt belongs downstream of evidence matching and upstream of human review routing. It assumes you have already run claim extraction, evidence retrieval, and claim-evidence pairing. It does not perform evidence retrieval, claim decomposition, or source authority assessment itself. The ideal input is a structured array of per-claim verification results with fields like verdict, evidence_count, source_authority_scores, contradiction_flags, and corroboration_count. If you feed this prompt raw documents or unpaired claims, it will hallucinate evidence connections. Wire it into your pipeline after your evidence matcher and before your triage router. For high-risk domains like healthcare or legal, always route outputs with confidence below a calibrated threshold to human review and log the full prompt input and output for audit trails.

Do not use this prompt when you need binary true/false labels without explanation, when you lack structured evidence-matching results, or when latency constraints prohibit the additional inference step of confidence calibration. It is also not a replacement for a statistical calibration layer—this prompt produces qualitative explanations and factor breakdowns, not mathematically calibrated probabilities. If you need Platt scaling or isotonic regression over model logprobs, implement that in application code and use this prompt only for the human-readable explanation layer. Start by validating your input schema against the expected fields, then run the prompt on a small batch with known outcomes to calibrate your confidence thresholds before scaling to production volumes.

This prompt template generates a structured Verification Confidence Report from a set of pre-verified claims and their associated evidence. It is designed for verification system architects who need calibrated confidence scores with explanations, not just binary true/false labels. The prompt instructs the model to produce a report detailing confidence levels per claim, factors influencing confidence, and calibration notes. Use this as the core instruction block in your verification pipeline, replacing square-bracket placeholders with your actual data before each run.

textCopy
You are an expert verification analyst producing calibrated confidence reports. Your task is to generate a structured Verification Confidence Report from the provided claims and evidence.

## INPUT DATA
Claims and Evidence:
[CLAIMS_AND_EVIDENCE_JSON]

## OUTPUT SCHEMA
Return a single JSON object conforming to this structure:
{
  "report_id": "string",
  "generated_at": "ISO 8601 timestamp",
  "claims": [
    {
      "claim_id": "string",
      "claim_text": "string",
      "verdict": "SUPPORTED | CONTRADICTED | UNVERIFIED | DISPUTED",
      "confidence_score": 0.0-1.0,
      "confidence_level": "HIGH | MEDIUM | LOW",
      "calibration_notes": "Explanation of factors affecting this confidence score, including evidence quality, source agreement, and any uncertainty drivers.",
      "overconfidence_risk": "LOW | MEDIUM | HIGH",
      "overconfidence_rationale": "Specific reasons why this score might be overconfident, such as limited source diversity, outdated evidence, or ambiguous language.",
      "evidence_summary": [
        {
          "source_id": "string",
          "relevance": "DIRECT | INDIRECT | TANGENTIAL",
          "impact": "SUPPORTS | CONTRADICTS | NEUTRAL"
        }
      ]
    }
  ],
  "aggregate_metrics": {
    "total_claims": "integer",
    "high_confidence_count": "integer",
    "medium_confidence_count": "integer",
    "low_confidence_count": "integer",
    "overconfidence_risk_distribution": {
      "high": "integer",
      "medium": "integer",
      "low": "integer"
    }
  },
  "calibration_warnings": ["List of systemic issues detected, such as 'Evidence set lacks authoritative sources for financial claims' or 'Multiple claims rely on single-source evidence.'"]
}

## CONSTRAINTS
- Assign confidence scores based on evidence quality, source authority, corroboration count, and contradiction presence.
- Never assign HIGH confidence to claims supported by a single source unless that source is explicitly marked as authoritative for the domain.
- Flag overconfidence risk as HIGH when evidence is thin, sources are low-authority, or the claim involves prediction or interpretation.
- Include specific, actionable calibration notes for every claim. Do not use generic phrases like "evidence supports claim."
- If a claim lacks sufficient evidence to assign a verdict, set verdict to UNVERIFIED and confidence_level to LOW.
- Do not invent evidence or sources. Only reference sources provided in the input.

## RISK_LEVEL
[RISK_LEVEL]

## EXAMPLES
[FEW_SHOT_EXAMPLES]

After pasting this template, replace [CLAIMS_AND_EVIDENCE_JSON] with your structured input containing claim objects and their associated evidence records. Set [RISK_LEVEL] to HIGH if the output will be used in regulated or compliance contexts—this adds stricter calibration language and requires explicit uncertainty disclosure. Populate [FEW_SHOT_EXAMPLES] with 2-3 example claim-evidence pairs and their ideal confidence reports to anchor the model's behavior. Always validate the output JSON against the schema before downstream consumption. For high-risk domains, route reports with HIGH overconfidence risk to human review before publication.

The Verification Confidence Report prompt is designed to be the final scoring stage in a multi-step verification pipeline, not a standalone chat interaction. It expects pre-extracted claims and pre-matched evidence as input, meaning the application layer must already have completed claim decomposition and evidence retrieval before this prompt is called. The prompt produces a structured confidence report per claim, which downstream systems can use for triage routing, audit logging, or reviewer dashboards. Wiring it correctly requires a clear contract between the upstream pipeline stages and this scoring stage, with explicit schema validation on both the input payload and the output report.

In a typical implementation, the application calls this prompt inside a processing loop that iterates over a batch of verified claims. Each iteration assembles the prompt with the claim text, the matched evidence records, source metadata, and any domain-specific calibration parameters. The model response must be parsed as JSON and validated against the expected output schema before it enters the audit trail. Key validation checks include: every claim in the input batch has a corresponding confidence report entry, confidence scores fall within the declared range (e.g., 0.0 to 1.0), every factor influencing confidence is explicitly listed with a weight or explanation, and no report entry is missing required fields such as claim_id, confidence_score, or calibration_notes. Failed validation should trigger a retry with the validation error message appended to the prompt as additional context, up to a configured maximum retry count (typically 2-3 attempts). After the retry budget is exhausted, the claim should be routed to a human review queue with the partial output and validation failure details attached.

Model choice matters for confidence calibration. Use a model that supports structured output with low temperature (0.0-0.1) to reduce variance in confidence scores across repeated runs. If the model supports logprobs, log the token-level probabilities for the confidence score field to enable downstream calibration analysis. For high-stakes domains such as healthcare or legal verification, implement a confidence threshold gate: claims scoring below a configurable threshold (e.g., 0.7) are automatically escalated for human review regardless of validation success. Log every prompt invocation, the raw model response, validation results, retry count, and final disposition to an observability system. This trace data is essential for detecting overconfidence drift, calibration decay, and systematic scoring errors over time. Avoid using this prompt in real-time user-facing flows without a human-in-the-loop fallback; confidence reports are advisory and should not be treated as final verdicts without review in regulated or high-risk contexts.