Confidence-Aware Summarization Prompt

This prompt is designed for summarization product teams who need to surface uncertainty directly in the output, not hide it in a separate metadata field. The core job-to-be-done is producing a summary where every factual claim is tagged with an inline confidence marker—such as [HIGH], [MEDIUM], or [LOW]—so that downstream systems can filter, highlight, or route low-confidence claims for human review. The ideal user is a product engineer or AI architect building a feature where fact-level precision and recall against source material matter more than prose fluency. Required context includes the source document, a clear definition of what constitutes a factual claim, and a confidence taxonomy that your application can act on programmatically.

Use this prompt when your product requires users to distinguish between well-supported statements and the model's inferences, when you have a review queue or UI that can consume the markers, and when your evaluation pipeline measures span-level accuracy against ground-truth annotations. The inline marker approach works well for legal document review, clinical note summarization, financial report digestion, and any workflow where a single hallucinated claim carries disproportionate risk. You should also use this when you need to generate training data for a confidence classifier or when you are building a human-in-the-loop system that routes only low-confidence spans for expert verification rather than reviewing entire summaries.

Do not use this prompt for latency-sensitive streaming summaries where inline markers would disrupt the reading experience, for consumer-facing chat summaries where users expect fluent prose without annotation noise, or when your downstream UI cannot parse or display the markers. It is also the wrong choice when the source material is too short to contain multiple factual claims, when your confidence taxonomy is poorly defined and leads to inconsistent marker assignment, or when you lack a validation pipeline to measure whether the markers actually correlate with factual accuracy. In those cases, consider a separate confidence score output or a structured abstention prompt instead. Before deploying, run span-level accuracy checks against human-annotated confidence labels and establish a calibration baseline so you know whether the model's self-assessed confidence tracks reality.

This prompt template is the core instruction set for generating a confidence-aware summary. It forces the model to separate factual claims from interpretive language, anchor each claim to the source material, and assign a confidence level based on how directly the source supports the claim. The output is a structured summary with inline markers that downstream systems can parse for filtering, highlighting, or escalation decisions. Use this template when you need to surface uncertainty to end users or when a downstream process—such as a review queue or a fact-verification pipeline—requires claim-level confidence signals before acting on the summary.

textCopy
You are a precision summarization system. Your task is to produce a summary of the provided source material with inline confidence markers on every factual claim.

## INPUT
Source Material:
[SOURCE_TEXT]

Summary Length Target: [LENGTH_CONSTRAINT]
Audience and Purpose: [AUDIENCE_CONTEXT]

## OUTPUT SCHEMA
Return a JSON object with the following structure:
{
  "summary_uid": "string, unique identifier for this summary run",
  "source_hash": "string, hash of the source text for traceability",
  "claims": [
    {
      "claim_id": "string, unique within this summary",
      "claim_text": "string, a single factual assertion extracted from or implied by the source",
      "claim_type": "factual_assertion | interpretation | numerical_fact | temporal_fact | causal_claim",
      "source_span": "string, the exact text span from the source that supports this claim, or null if unsupported",
      "confidence": "high | medium | low | unsupported",
      "confidence_rationale": "string, brief explanation of why this confidence level was assigned",
      "is_verifiable": true or false
    }
  ],
  "summary_text": "string, a fluent narrative summary that incorporates all claims",
  "overall_confidence": "high | medium | low",
  "uncertainty_flags": ["string, list of global uncertainty factors affecting this summary"]
}

## CONFIDENCE DEFINITIONS
- **high**: The source explicitly states this claim with no ambiguity. The source_span is a direct quote or near-direct quote.
- **medium**: The source strongly implies this claim, or the claim requires combining two pieces of information from the source. The source_span provides substantial but not verbatim support.
- **low**: The claim is a reasonable inference from the source but is not directly stated. The source_span provides partial or circumstantial support.
- **unsupported**: The claim cannot be grounded in the source material. source_span must be null. Include these only if the claim is necessary for summary coherence; otherwise omit.

## CONSTRAINTS
1. Every factual assertion in summary_text must appear as a separate entry in the claims array.
2. Do not fabricate facts. If the source does not contain information needed for a coherent summary, flag this in uncertainty_flags rather than inventing content.
3. For each claim, extract the minimal source_span that supports it. Do not paraphrase the source_span; quote it exactly.
4. If two claims in the source conflict, include both, assign appropriate confidence levels, and note the conflict in uncertainty_flags.
5. If the source contains numerical data, preserve precision. Do not round or approximate unless the source does.
6. If the source is too short or too long to summarize effectively, include a flag in uncertainty_flags and produce the best summary possible with appropriate confidence downgrades.
7. Do not include claims that are common knowledge unless they appear in the source.

## EXAMPLES
[FEW_SHOT_EXAMPLES]

## RISK LEVEL
[RISK_LEVEL: low | medium | high | critical]
- If RISK_LEVEL is high or critical, prefer lower confidence assignments when uncertain and include more detailed confidence_rationale entries.
- If RISK_LEVEL is low, you may assign medium confidence to reasonable inferences without exhaustive justification.

Adapt this template by adjusting the confidence definitions to match your domain's tolerance for uncertainty. For regulated domains, add a review_required boolean field to each claim and set it to true when confidence is low or unsupported. Replace [FEW_SHOT_EXAMPLES] with 2–3 annotated examples that show the boundary between medium and low confidence for your specific content type. The [RISK_LEVEL] parameter should be wired to a runtime configuration that your application sets based on the use case—customer-facing summaries might use medium, while audit-facing summaries should use high or critical. After pasting this template, test it against source material where you know the ground-truth claims, and measure both claim recall (did the model extract all important facts?) and confidence calibration (do high-confidence claims match verifiable facts at the expected rate?).

The Confidence-Aware Summarization Prompt is designed to be called from an application harness, not used as a one-off chat. The harness is responsible for providing the source document, managing the model request, parsing the structured output, and acting on the confidence markers. The prompt returns a JSON object containing a summary string and a claims array, where each claim includes the claim text, a confidence score (0.0–1.0), and a source_span reference. The application should treat this output as a machine-readable signal, not a final rendered artifact. Downstream consumers—such as a UI highlighter, a fact-verification pipeline, or a human review queue—will consume the claims array to make filtering, display, or escalation decisions.

To integrate this prompt, wrap the call in a function that validates the response against a strict schema before any downstream use. Use a JSON Schema validator to confirm that every claim object contains a numeric confidence between 0 and 1, a non-empty claim string, and a source_span that maps to the provided source text. If validation fails, retry once with a repair prompt that includes the original input, the malformed output, and the specific validation error. Log every validation failure and retry attempt for observability. For model selection, prefer models with strong instruction-following and JSON mode support (e.g., GPT-4o, Claude 3.5 Sonnet) and set response_format to json_object or use constrained decoding where available. If the model does not support strict JSON mode, increase the retry budget and add a regex-based extraction fallback before declaring failure.

The confidence scores are model self-assessments and should not be treated as calibrated probabilities without evaluation. Run an offline calibration check by comparing claim-level confidence scores against human-annotated factual accuracy labels on a golden dataset. If the model is overconfident (high confidence on incorrect claims) or underconfident (low confidence on correct claims), adjust the threshold for downstream actions or add a calibration layer. In high-stakes deployments—such as medical or legal summarization—route any summary where more than a configurable percentage of claims fall below a confidence threshold (e.g., 0.7) to a human review queue. The harness should attach the original document, the full claims array, and a pre-formatted review prompt to the escalation payload. Do not display low-confidence claims as fact to end users without visual uncertainty indicators or explicit caveats.