Rubric for Factual Accuracy Scoring Prompt Template

This prompt is designed for a specific, high-stakes job: turning a source document and an AI-generated output into a structured, repeatable factuality score. The ideal user is an evaluation engineer or a verification team lead who already has a 'known-correct' reference text and needs to programmatically determine if a summary, answer, or report faithfully represents it. The core value is in producing a detailed rubric that an LLM judge can apply consistently, moving factuality assessment from subjective human spot-checks to an automated, auditable process. This is not a general-purpose quality prompt; it is a precision instrument for a single dimension of quality: factual adherence to a provided source.

You should use this prompt when you have a clear ground-truth evidence set and need to score an output for factuality. This is common in RAG system evaluation, document summarization QA, and any workflow where an AI synthesizes information from a provided text. The prompt is built to handle nuance, including partial credit for statements that are mostly correct but have minor errors, and it classifies the severity of hallucinations. However, you must avoid using this prompt when the evaluation criteria are subjective, such as grading the 'creativity' of a story, the 'persuasiveness' of an argument, or the 'tone' of an email. It is strictly for measuring what is true, what is missing, and what is invented, based solely on the provided evidence.

Before integrating this prompt into your evaluation pipeline, ensure your ground-truth documents are well-structured and your AI outputs are scoped to the content of those documents. The prompt's effectiveness is entirely dependent on the quality and completeness of the [GROUND_TRUTH] you provide. A common failure mode is feeding it an output that requires external world knowledge not present in the evidence, which will lead to false positives for hallucination. The next step is to copy the prompt template and begin adapting the [OUTPUT_SCHEMA] and [SEVERITY_SCALE] to match your specific tolerance for error. Do not deploy this as a final gate without first running a calibration exercise against human evaluators to ensure the rubric's definitions of 'minor' and 'major' hallucination align with your team's standards.

This prompt template is designed to act as an LLM judge for factual accuracy. It instructs the model to extract verifiable claims from a [TARGET_TEXT], compare them against provided [GROUND_TRUTH_EVIDENCE], and produce a structured scorecard. The rubric includes rules for claim extraction, evidence matching, partial credit for nuanced statements, and a severity classification for any hallucinations found. Use this as the core instruction set for an automated evaluation pipeline.

markdownCopy
You are a meticulous Factual Accuracy Judge. Your task is to evaluate the factual accuracy of the provided [TARGET_TEXT] against the [GROUND_TRUTH_EVIDENCE]. Follow these steps precisely and output the results in the specified [OUTPUT_SCHEMA].

### Step 1: Claim Extraction
1.  Decompose the [TARGET_TEXT] into a list of discrete, verifiable factual claims.
2.  A claim must be a single, self-contained statement that can be proven true or false.
3.  Ignore opinions, stylistic flourishes, and subjective statements unless they imply a verifiable fact.
4.  Output each claim with a unique `claim_id`.

### Step 2: Evidence Matching and Verification
For each extracted claim, search the [GROUND_TRUTH_EVIDENCE] to find supporting or contradicting information.
- **Supported:** The claim is directly and unambiguously confirmed by the evidence. Provide the exact quote from the evidence.
- **Contradicted:** The claim is directly and unambiguously disproven by the evidence. Provide the exact quote from the evidence.
- **Partially Supported:** The claim contains a mix of accurate and inaccurate information, or the evidence only confirms a portion of it. Explain the discrepancy.
- **Unverifiable:** The claim cannot be confirmed or denied based on the provided evidence. Do not use external knowledge.

### Step 3: Scoring
Assign a score to each claim based on the verification result:
- **1 (Accurate):** Claim is Supported.
- **0.5 (Partially Accurate):** Claim is Partially Supported.
- **0 (Inaccurate):** Claim is Contradicted.
- **NULL (Unverifiable):** Claim is Unverifiable. Exclude NULL scores from the final accuracy calculation.

### Step 4: Hallucination Severity Classification
For any claim scored as 0 (Inaccurate) or 0.5 (Partially Accurate), classify the severity of the error:
- **Critical:** The error introduces dangerous, harmful, or legally/financially damaging misinformation.
- **Major:** The error significantly alters the meaning or introduces a key false fact.
- **Minor:** The error is a small, non-material inaccuracy (e.g., a slightly wrong date, a misspelled name that doesn't change the entity).

### Step 5: Final Score Calculation
Calculate the overall factual accuracy score as: `(Sum of scores for all verifiable claims) / (Total number of verifiable claims)`. A verifiable claim is any claim not scored as NULL.

### Input Data
<TARGET_TEXT>
[TARGET_TEXT]
</TARGET_TEXT>

<GROUND_TRUTH_EVIDENCE>
[GROUND_TRUTH_EVIDENCE]
</GROUND_TRUTH_EVIDENCE>

### Output Schema
You must output a single JSON object matching this exact schema:
{
  "overall_factual_accuracy_score": "number",
  "total_verifiable_claims": "integer",
  "claim_verifications": [
    {
      "claim_id": "string",
      "claim_text": "string",
      "verification_result": "Supported | Contradicted | Partially Supported | Unverifiable",
      "evidence_quote": "string or null",
      "score": "number (1, 0.5, 0, or null)",
      "severity": "Critical | Major | Minor | null",
      "judge_rationale": "string"
    }
  ],
  "overall_summary": "string"
}

To adapt this template, replace the [TARGET_TEXT] and [GROUND_TRUTH_EVIDENCE] placeholders with your actual data. For high-stakes domains like healthcare or finance, the severity classification becomes critical for downstream routing. You can adjust the scoring weights or add new verification results like "Opinion" if your use case requires it. The key to reliable output is enforcing the [OUTPUT_SCHEMA] in your application's API call or parser, and always logging the judge_rationale field to debug scoring inconsistencies.

This prompt is designed to be called programmatically as part of an automated evaluation pipeline, not as a one-off manual check. The typical integration pattern places this prompt after an LLM has generated a response and after a retrieval step has provided source evidence. The harness should pass the generated text as [RESPONSE], the source documents as [SOURCE_EVIDENCE], and any domain-specific factuality constraints as [DOMAIN_CONSTRAINTS]. The output is a structured JSON object containing extracted claims, evidence matches, partial-credit assignments, and hallucination severity classifications. This output should be parsed by the application layer, not left as raw text for human interpretation.

Validation and retry logic is critical because the prompt produces a complex nested schema. Implement a JSON schema validator that checks for required fields (claims, overall_score, hallucination_severity), correct enum values for severity levels (none, minor, moderate, major, critical), and numeric score ranges (0.0 to 1.0). If validation fails, retry the prompt once with the validation error message appended to the [CONSTRAINTS] field, instructing the model to fix the schema violation. If the second attempt also fails, log the raw output and route the sample to a human review queue rather than silently accepting a malformed score. For high-stakes domains like healthcare or legal review, set [RISK_LEVEL] to high to trigger mandatory human verification of any output where hallucination_severity is major or critical.

Model choice and latency considerations matter for this prompt because claim extraction and evidence matching are reasoning-intensive tasks. Use a capable model (GPT-4, Claude 3.5 Sonnet, or equivalent) for the primary scoring pass. For cost optimization on large evaluation batches, consider a two-stage architecture: use a faster, cheaper model to extract claims and flag potential issues, then route only flagged or borderline samples to the full rubric prompt. Store all outputs with prompt_version, model_id, timestamp, and source_document_ids for auditability. When integrating with RAG systems, ensure [SOURCE_EVIDENCE] includes document chunk IDs so the evaluation output can trace each claim back to its supporting or contradicting source. Avoid using this prompt on streaming outputs—always score complete, finalized responses to prevent partial-claim extraction errors.