Low-Confidence Citation Flagging Prompt

This prompt is designed for AI product teams operating cited answer systems where every claim must be traceable to a source, but not every source carries equal weight. The primary job-to-be-done is to programmatically identify citations that fall below a defined confidence threshold and prepare them for human review without disrupting the answer's structure or delaying low-risk outputs. The ideal user is a platform engineer or AI reliability engineer building a human-in-the-loop review queue. They need a prompt that takes a generated answer, its inline citations, and per-citation confidence metadata, then returns a structured flagging report that a review UI can consume directly.

Use this prompt when your system already assigns confidence scores to individual citations—whether from retrieval relevance, source authority, claim-to-passage alignment distance, or a composite model-graded metric. The prompt assumes you have a threshold below which a citation is considered unreliable. It does not generate confidence scores; it acts on them. Required context includes the full answer text with citation markers, a mapping of citation IDs to source passages, and a confidence score per citation. The output should be a machine-readable list of flagged citations, each with the reason for flagging, the surrounding claim text, and clear review instructions for a human operator.

Do not use this prompt as a substitute for upstream confidence scoring. If your system lacks per-citation scores, start with the Citation Confidence Scoring Prompt or the Source-to-Claim Alignment Verification Prompt in this content group. Do not use it for real-time user-facing corrections—this is a review-queue workflow, not a live regeneration trigger. In regulated domains such as healthcare, legal, or finance, always route flagged outputs to a qualified human reviewer and log the flagging decision with the full evidence chain. The prompt is a triage tool, not a compliance guarantee.

This template is designed to be dropped into a human-in-the-loop review pipeline. It takes a generated answer with inline citations, their associated confidence scores, and a set of flagging criteria, then produces a marked-up version of the answer that highlights which citations need human verification. The prompt does not alter the answer's semantic content—it only adds flags and review instructions so that a human reviewer can quickly focus on the citations that are most likely to be wrong, hallucinated, or misaligned with their source passages.

textCopy
You are a citation quality auditor. Your task is to review a generated answer that contains inline citations, identify which citations have low confidence scores according to the provided criteria, and flag them for human review. Do not modify the answer text itself. Preserve all original wording, structure, and citations exactly as provided.

## INPUT

**Answer with citations:**
[ANSWER_WITH_CITATIONS]

**Citation confidence scores:**
[CITATION_CONFIDENCE_SCORES]

**Flagging criteria:**
[FLAGGING_CRITERIA]

**Flag format:**
[FLAG_FORMAT]

## INSTRUCTIONS

1. Parse the answer to identify every inline citation marker.
2. For each citation, look up its confidence score in the provided scores.
3. Apply the flagging criteria to determine whether the citation should be flagged.
4. For citations that meet the flagging criteria, insert a review flag immediately after the citation marker using the specified flag format. The flag must include:
   - The citation identifier
   - The confidence score
   - The specific criterion that triggered the flag
   - A brief note on what the human reviewer should check (e.g., "verify quote accuracy", "confirm source exists", "check claim alignment")
5. For citations that do NOT meet the flagging criteria, leave them unchanged.
6. Output the complete answer with flags inserted. Do not add any commentary, preamble, or postamble outside the answer text.

## CONSTRAINTS

- Do not modify, rewrite, or paraphrase any part of the original answer.
- Do not remove or relocate any citation markers.
- Do not add citations that were not present in the original answer.
- Do not change the order of sentences or paragraphs.
- If no citations meet the flagging criteria, return the answer completely unchanged.
- If a citation appears multiple times, flag each occurrence independently.

## OUTPUT FORMAT

Return the complete answer text with review flags inserted inline. Use the exact flag format specified in the input. Do not wrap the output in JSON, markdown fences, or any other container.

To adapt this template, replace the square-bracket placeholders with your pipeline's specific inputs. [ANSWER_WITH_CITATIONS] should contain the full generated text with citation markers (e.g., [1], [src-4]). [CITATION_CONFIDENCE_SCORES] should be a structured mapping of citation identifiers to scores, such as {"1": 0.92, "2": 0.34, "3": 0.78}. [FLAGGING_CRITERIA] defines your thresholds and rules—for example, "flag any citation with confidence below 0.6" or "flag citations where source authority score is below 0.5 AND relevance score is below 0.7." [FLAG_FORMAT] specifies the exact markup to insert, such as [REVIEW NEEDED: citation {id}, confidence {score}, reason: {criterion}]. If your pipeline uses structured outputs, you can replace the free-text flag format with a JSON annotation schema and add an [OUTPUT_SCHEMA] placeholder to enforce it. For high-stakes domains like healthcare or legal review, add a [RISK_LEVEL] placeholder that adjusts the flagging threshold downward and requires more explicit reviewer instructions in each flag.

The Low-Confidence Citation Flagging Prompt is designed to sit between your RAG answer generator and a human review queue. It should not be called on every output—only when upstream confidence signals (retrieval scores, claim-to-source alignment distances, or model self-assessment tokens) fall below defined thresholds. Wire this prompt as a gating step: if the flagging prompt returns zero low-confidence citations, the answer proceeds to the user. If it returns one or more flagged citations, the answer and its evidence are routed to a review interface with the flagging metadata attached.

Implement the prompt inside a structured validation harness. The model must return a JSON object with a flagged_citations array, where each entry includes the citation index, the confidence score, the flagging reason, and the suggested review action. Validate this schema before routing. If the model returns malformed JSON, retry once with a stricter output constraint appended to the prompt. If the retry also fails, escalate the entire answer to human review with a schema-failure marker. Log every flagging result—including zero-flag outcomes—so you can track flagging rates, reviewer overturn ratios, and confidence score distributions over time.

Model choice matters here. Use a model with strong instruction-following and structured output capabilities for the flagging step. The flagging prompt itself is a classification and annotation task, not a generation task, so prioritize precision over creativity. Set temperature to 0 or near-zero. If your system already computes retrieval scores or alignment distances, pass those as pre-computed fields in the [CITATION_METADATA] input rather than asking the model to re-derive them. This reduces hallucination risk and keeps the model focused on threshold comparison and flagging rationale. For high-stakes domains, never auto-accept a zero-flag result without a periodic audit sample going to human review anyway—drift in flagging behavior is a leading indicator of prompt degradation.

Wire the output into your review queue with clear triage fields. Each flagged citation should carry a review_priority enum (high, medium, low) derived from the confidence score and the claim's materiality. High-priority flags (score below 0.5 on a 0–1 scale, or claims affecting compliance, safety, or financial figures) should surface at the top of the review queue. The review interface should display the original answer, the flagged citation inline, the source passage, and the model's flagging reason. After human review, capture the reviewer's decision (accept, correct, reject) and feed it back into your eval dataset for ongoing prompt calibration. Avoid the temptation to auto-correct based on the flagging prompt alone—this prompt identifies risk, it does not resolve it.