Claim-by-Claim Faithfulness Audit Prompt Template

This prompt is for evaluation engineers and AI quality teams who need to verify whether a generated answer is fully supported by the provided evidence. Use it when you need a structured, claim-level audit that decomposes an answer into atomic factual statements and checks each one independently against source material. The ideal user is someone operating a RAG pipeline, summarization system, or content generation workflow who already has a candidate answer and a set of evidence passages and needs a machine-readable audit report before that answer reaches a user or a downstream system.

This is not a prompt for generating answers. It is a verification prompt that sits downstream from your generation step. You should use it when the cost of an unsupported claim is high—for example, in regulated domains, customer-facing factual products, or any system where hallucination directly erodes user trust. It is also the right tool when you need to calibrate human annotators, run regression tests on a golden dataset, or feed a production guardrail that blocks or flags unfaithful outputs. The prompt assumes you have already retrieved evidence and generated an answer; it does not perform retrieval or generation itself.

Do not use this prompt when you need a single holistic faithfulness score without claim-level granularity—use a rubric-based LLM judge instead. Do not use it when your evidence set is empty or when the answer is purely subjective, stylistic, or conversational with no factual claims to audit. Avoid this prompt if your latency budget is extremely tight and you cannot afford an extra model call per answer; in those cases, consider a lighter-weight classifier or a streaming guardrail. Finally, do not use this prompt as a substitute for human review in high-stakes decisions where regulatory or safety requirements mandate a person in the loop. The output is a structured audit, not a legal certification.

Before integrating this prompt, ensure you have a clear definition of what constitutes a 'claim' in your domain and a consistent evidence format. The prompt works best when evidence passages are self-contained and have stable identifiers. If your evidence is messy, truncated, or lacks provenance metadata, the audit quality will degrade. Start by running this prompt against a labeled calibration set to establish baseline agreement with human annotators, then integrate it into your evaluation harness or guardrail pipeline.

This template is the core instruction set for the Claim-by-Claim Faithfulness Audit. It instructs the model to act as an evaluation auditor, decompose a generated answer into discrete factual claims, and check each claim against a provided evidence set. The prompt is designed to be copied directly into your evaluation harness, with square-bracket placeholders replaced by your application's runtime values before each call.

textCopy
You are an evaluation auditor performing a claim-by-claim faithfulness audit. Your task is to decompose a generated answer into individual factual claims and verify each claim against the provided evidence.

## INPUT

**Generated Answer:**
[ANSWER]

**Evidence Set:**
[EVIDENCE]

**Audit Configuration:**
- Claim granularity: [GRANULARITY]
- Minimum support threshold: [SUPPORT_THRESHOLD]
- Output schema: [OUTPUT_SCHEMA]

## INSTRUCTIONS

1. **Decompose** the generated answer into discrete, atomic factual claims. Each claim must be a single verifiable statement. Do not combine multiple facts into one claim.

2. For each claim, **search the evidence set** for direct support, partial support, or contradiction. A claim is:
   - **FULLY_SUPPORTED**: The evidence contains explicit information that confirms the claim.
   - **PARTIALLY_SUPPORTED**: The evidence supports some aspects but not all, or supports a weaker version of the claim.
   - **UNSUPPORTED**: No relevant evidence exists for this claim.
   - **CONTRADICTED**: The evidence directly conflicts with the claim.

3. For each verdict, **cite the exact evidence span** that supports your determination. If no evidence exists, state "No relevant evidence found."

4. **Do not infer** support from related but distinct facts. If the evidence says "the system processes 500 requests per second" and the answer claims "the system handles 500 concurrent users," mark it UNSUPPORTED unless the evidence explicitly equates requests with users.

5. If the evidence set is empty or insufficient to evaluate any claims, return a single audit entry with verdict INSUFFICIENT_EVIDENCE and explain what is missing.

## OUTPUT FORMAT

Return a JSON object matching the schema below. Do not include commentary outside the JSON.

[OUTPUT_SCHEMA]

## CONSTRAINTS

[CONSTRAINTS]

Replace each placeholder before sending the prompt to the model. [ANSWER] should contain the full generated text under audit. [EVIDENCE] should contain the retrieved passages, documents, or source material formatted as plain text with clear source identifiers. [GRANULARITY] controls how finely the model splits claims—use sentence-level for coarse audits or atomic-fact for strict verification. [SUPPORT_THRESHOLD] defines the minimum evidence match required for FULLY_SUPPORTED, typically exact-match or semantic-equivalence. [OUTPUT_SCHEMA] should be a JSON schema defining the expected structure, such as an array of objects with claim_id, claim_text, verdict, evidence_citation, and rationale fields. [CONSTRAINTS] can include domain-specific rules, such as "Do not consider evidence from sources older than 2023" or "Mark any claim involving financial figures as requiring human review if partially supported."

Before deploying this template into a production evaluation pipeline, test it against a golden dataset of answer-evidence pairs with known faithfulness labels. Verify that the model's verdicts align with human annotators and that the decomposition step does not miss compound claims. For high-risk domains, always route CONTRADICTED and UNSUPPORTED verdicts to a human review queue before surfacing results to end users or downstream systems.

The claim-by-claim faithfulness audit prompt is designed to operate inside a structured evaluation harness, not as a one-off chat interaction. In production, you will typically run this prompt over hundreds or thousands of answer-evidence pairs, collect structured audit results, and aggregate verdicts into dashboards, regression gates, or inter-annotator calibration reports. The harness is responsible for feeding each pair through the prompt, parsing the JSON output, validating the schema, handling malformed responses, and recording per-claim verdicts for downstream analysis.

Start by defining a batch input schema: each record should contain a unique audit_id, the generated_answer text, and the evidence_passages array with passage_id and content fields. Wrap the prompt call in a retry loop with exponential backoff (3 attempts minimum) and a JSON schema validator that checks for required fields: claim_id, claim_text, verdict (one of supported, partially_supported, unsupported, contradicted), evidence_citation (array of passage_id and quote), and rationale. If validation fails, feed the raw response and the schema errors into a repair prompt that asks the model to fix the output structure without changing the verdicts. Log every raw response, parse failure, and repair attempt for traceability. For high-stakes audits in regulated domains, route all unsupported and contradicted verdicts to a human review queue before accepting them into the final report.

Model choice matters for this workflow. Use a model with strong instruction-following and JSON output discipline—GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro are reasonable defaults. Avoid smaller or older models that drift toward narrative summaries instead of structured claim decomposition. Set temperature=0 for deterministic audits and response_format to JSON mode if the model supports it. For batch evaluation at scale, parallelize calls across multiple API keys or use a batch API endpoint to reduce cost and latency. Store results in a database with columns for audit_id, claim_id, verdict, evidence_citation, rationale, model_version, prompt_version, and timestamp so you can track drift across prompt iterations and model upgrades. Before shipping a new prompt version, run a regression test against a golden dataset of 50-100 annotated answer-evidence pairs and confirm that verdict agreement with human labels does not degrade.