RAG Citation Answer with Confidence Score Prompt Template

This prompt is designed for RAG system builders who need every factual claim in an answer to be linked to a specific source chunk with a confidence score. The primary job-to-be-done is generating a synthesized answer from multiple retrieved chunks while explicitly rating how well each claim is supported by the evidence. The ideal user is an engineering lead or AI/ML engineer building a production Q&A system where audit, compliance, or user trust requires traceable citations. Required context includes a set of retrieved candidate chunks with identifiers, a user question, and a defined confidence scale. Without this context, the prompt cannot produce grounded, scored citations.

Use this prompt when your retrieval pipeline returns multiple candidate chunks and you need the model to synthesize an answer while attaching a confidence indicator to each source. It is particularly effective for compliance review systems, internal knowledge base Q&A, and customer-facing assistants where users need to verify answers against source material. The confidence score should be treated as a signal for downstream routing: high-confidence claims can be displayed directly, medium-confidence claims should be flagged for review, and low-confidence claims should be suppressed or re-queried. Pair this prompt with a Citation Completeness Gate to block outputs where material claims lack citations, and a Hallucination Detection via Citation Verification prompt to validate that cited sources actually support the claims made.

Do not use this prompt when you need exact text spans, page-line coordinates, or byte-level offsets. For those requirements, pair it with the Citation Span Detection and Boundary prompt. Avoid this prompt when your retrieval pipeline returns a single monolithic document without chunk identifiers, as the model will have no way to differentiate sources. Also avoid it when the user question requires subjective analysis or opinion rather than factual synthesis from provided evidence. In high-risk domains such as healthcare or legal review, always route outputs through human approval before they reach end users. The confidence scores this prompt produces are model-generated estimates, not calibrated probabilities, and should be validated against human judgments using the Source Grounding Score prompt before relying on them for automated decisions.

This template is designed to be dropped directly into your RAG pipeline. It forces the model to decompose its answer into individual claims, link each claim to a specific source chunk, and assign a confidence score based on how explicitly the source supports the claim. The output schema is strict JSON, making it suitable for programmatic validation before the answer reaches a user interface. Use this template when you need auditability, when downstream systems consume structured citation data, or when you must detect and suppress unsupported claims before they appear in production.

textCopy
You are a precise research assistant. Answer the user question using ONLY the provided source chunks. For every factual claim in your answer, cite the source chunk ID that supports it and assign a confidence score from 0.0 to 1.0. A score of 1.0 means the source explicitly states the claim. A score of 0.5 means the source implies or partially supports the claim. A score of 0.0 means no source supports the claim and you must mark it as UNSUPPORTED. If no source chunk provides sufficient evidence, state that you cannot answer and explain what is missing.

[OUTPUT_SCHEMA]

[SOURCE_CHUNKS]

[QUESTION]

Adaptation guidance: Replace [OUTPUT_SCHEMA] with a concrete JSON schema that includes fields for claim, source_chunk_id, confidence_score, and supporting_quote. Replace [SOURCE_CHUNKS] with your retrieved chunks formatted as [CHUNK_ID: chunk_text]. Replace [QUESTION] with the user's query. For high-risk domains such as healthcare or legal, add a [CONSTRAINTS] block that explicitly instructs the model to abstain when confidence falls below 0.7 and to flag any claim that relies on inference rather than direct statement. Always validate the output against your schema before surfacing it to users, and log any claims with a confidence score below your production threshold for human review.

This prompt is designed to sit between your retrieval system and the final response renderer. After your retriever fetches candidate chunks and your reranker selects the most relevant passages, assemble the prompt by injecting the user's question into the [QUESTION] placeholder and the retrieved context into the [RETRIEVED_CONTEXT] placeholder. Call the LLM with this assembled prompt and parse the JSON response into a structured object. The output schema expects an array of claim objects, each containing a claim_text, source_chunk_id, and confidence_score. Do not pass the raw LLM output directly to the user; always parse and validate it first.

Implement a validation layer immediately after parsing. Check that every claim object contains a non-empty source_chunk_id field and that confidence_score is a float between 0.0 and 1.0 inclusive. If any claim is marked as UNSUPPORTED in the source_chunk_id field, treat it as an abstention signal and never display it to the user as if it were sourced. If validation fails, construct a retry prompt by appending the specific validation error message to the original prompt and call the LLM once more. Do not retry more than once; if the second attempt also fails validation, log the failure and route the interaction to a fallback path, such as a static 'unable to generate a reliable answer' response or a human review queue.

For observability, log every claim_text, source_chunk_id, and confidence_score triple to your tracing system. Include the full prompt, the raw LLM response, the parsed output, and the validation result in each trace entry. This data is essential for debugging citation hallucinations, monitoring confidence drift over time, and auditing model behavior. Use structured logging so you can query for patterns like claims with high confidence but missing source IDs or claims where confidence dropped after a model version upgrade.

In high-stakes domains such as legal review, healthcare, or financial compliance, route any answer where the overall_confidence field falls below 0.7 to a human review queue. Do not display low-confidence answers directly to end users without review. The human reviewer should see the original question, the retrieved context, the generated claims, and the confidence scores to make an informed decision. This gate prevents overconfident low-evidence answers from reaching users in contexts where errors carry material consequences.

Store the full prompt, response, and validation result in an append-only trace store for audit and debugging. This trace store should be queryable by interaction ID, timestamp, and confidence score. For regulated environments, ensure the trace store meets your retention and access control requirements. When debugging citation failures, compare the logged source_chunk_id values against your retrieval index to verify that the cited chunks actually exist and contain the claimed evidence. Missing or mismatched chunk IDs are a strong signal of hallucinated citations.