Factual Consistency Against Reference Prompt Template

This prompt is designed for evaluation engineers and AI reliability teams who need to automate the comparison of a model-generated output against a known-correct reference answer. The core job-to-be-done is producing a structured, machine-readable consistency score with span-level annotations that decompose the model output into atomic claims, align each claim to the reference, and flag unsupported or contradictory statements. The ideal user is someone integrating this prompt into a CI/CD gate for regression testing, benchmark maintenance, or pre-release validation where an authoritative 'golden answer' exists and any deviation must be measured precisely, not just flagged as a binary pass/fail.

Use this prompt when you have a stable, curated set of reference answers and need diagnostic detail to debug model drift. It is particularly effective in domains where partial credit matters—for example, a model might get the core fact right but hallucinate a supporting detail, and you need to capture both signals. The prompt expects a structured [MODEL_OUTPUT] and a [REFERENCE_ANSWER] as inputs, and it returns a consistency score alongside a claim-by-claim audit trail. This output can be consumed directly by an evaluation harness to block a release if the consistency score drops below a configured threshold or if a critical hallucination is detected.

Do not use this prompt when no authoritative reference answer is available, when the reference itself is contested or ambiguous, or when you only need a simple binary pass/fail without diagnostic detail. It is also unsuitable for evaluating creative or open-ended generation where multiple valid outputs exist without a single ground truth. In those cases, consider a pairwise comparison or rubric-based judge instead. For high-risk domains such as healthcare or legal review, always route flagged contradictions to a human reviewer before taking action, and ensure the reference answers themselves are subject to expert validation.

This prompt template is the core of your factual consistency evaluation pipeline. It instructs the model to act as a meticulous fact-checker, comparing a generated [MODEL_OUTPUT] against a known-correct [REFERENCE_ANSWER]. The prompt is designed to produce a structured JSON report that includes an overall consistency score, a detailed breakdown of supported, unsupported, and contradicted claims, and specific text spans from the output that are in question. This structured output is essential for automated downstream processing, such as flagging hallucinations for human review or calculating aggregate precision and recall metrics across a test suite.

textCopy
You are an expert fact-checker evaluating the factual consistency of a model-generated output against a verified reference answer. Your task is to produce a structured analysis that can be used for automated quality assurance.

# INPUTS
[MODEL_OUTPUT]: The text to be evaluated.
[REFERENCE_ANSWER]: The verified, ground-truth text to compare against.
[EVALUATION_CONTEXT]: Any additional context or instructions for the evaluation.

# INSTRUCTIONS
1.  **Decompose**: Break down the [MODEL_OUTPUT] into a list of discrete, verifiable factual claims.
2.  **Compare**: For each claim, determine its relationship to the [REFERENCE_ANSWER]. Use the following labels:
    - **SUPPORTED**: The claim is directly stated in or can be logically inferred from the reference.
    - **UNSUPPORTED**: The claim introduces new information not present in the reference.
    - **CONTRADICTED**: The claim directly conflicts with information in the reference.
3.  **Annotate**: For each claim, provide the exact text span from the [MODEL_OUTPUT] that contains the claim.
4.  **Score**: Calculate an overall factual consistency score on a scale of 0.0 to 1.0, where 1.0 means all claims are fully supported. The formula is: `(Number of SUPPORTED claims) / (Total number of claims)`.
5.  **Flag**: If any claim is CONTRADICTED, set a high-priority `hallucination_flag` to `true`.

# CONSTRAINTS
- Do not penalize the [MODEL_OUTPUT] for omitting information that is present in the [REFERENCE_ANSWER]. Only evaluate the claims that are actually made.
- Treat synonymous phrasing as SUPPORTED.
- If the [MODEL_OUTPUT] is empty or nonsensical, return a score of 0.0 and an appropriate error message.

# OUTPUT_SCHEMA
You must respond with a single valid JSON object conforming to this schema:
{
  "consistency_score": <float 0.0-1.0>,
  "hallucination_flag": <boolean>,
  "claim_analysis": [
    {
      "claim_text": "<exact text span from model output>",
      "verdict": "<SUPPORTED|UNSUPPORTED|CONTRADICTED>",
      "rationale": "<brief explanation referencing the reference answer>"
    }
  ],
  "error_message": "<null or a string if the output could not be evaluated>"
}

To adapt this template, replace the square-bracket placeholders with your actual data. The [MODEL_OUTPUT] and [REFERENCE_ANSWER] are the core inputs. The [EVALUATION_CONTEXT] placeholder is optional but powerful; use it to inject domain-specific rules, such as "ignore differences in date formatting" or "the reference answer is the definitive source for all statistical figures." Before integrating this into a CI/CD pipeline, run it against a small, hand-labeled golden dataset to calibrate the consistency_score threshold that constitutes a "pass" for your specific use case. A score of 0.9 might be acceptable for a summarization task, while a score of 1.0 might be required for a regulated financial report.

This prompt is designed to be called programmatically as a judge model within a CI/CD or batch evaluation pipeline, not as a one-off chat interaction. The harness should treat the LLM call as a deterministic function: input a model output and a reference answer, receive a structured consistency score with span-level annotations. The primary integration points are the evaluation runner (which iterates over test cases), the prompt rendering layer (which injects [MODEL_OUTPUT] and [REFERENCE_ANSWER] into the template), and the result consumer (which stores scores, flags hallucinations, and triggers alerts).

Validation and retry logic is critical because the output schema includes nested objects and enum-constrained fields. Implement a JSON schema validator that checks for the presence of overall_score, claim_annotations array, and correct enum values for verdict (must be one of supported, partially_supported, contradicted, unsupported). If validation fails, retry up to 2 times with the error message appended to the prompt as a correction hint. If the third attempt still fails, log the raw output and escalate for human review. For high-stakes domains like healthcare or finance, configure the harness to require human sign-off on any output where overall_score falls below 0.8 or where contradicted verdicts appear.

Model choice matters for this task. Use a model with strong instruction-following and structured output capabilities (GPT-4o, Claude 3.5 Sonnet, or equivalent). Set temperature=0 to maximize reproducibility across runs. If you're processing thousands of comparisons, implement batching with asynchronous calls and a concurrency limit appropriate for your API tier. Store every evaluation result—including the raw prompt, model response, parsed score, and validation status—in a structured log or database table. This traceability is essential for debugging score drift, auditing judge performance, and defending evaluation results to stakeholders. Avoid running this prompt on outputs that exceed the judge model's context window; if the combined [MODEL_OUTPUT] and [REFERENCE_ANSWER] exceed ~60% of the context limit, chunk the content or summarize before comparison.