Faithfulness Fidelity Score Rubric Prompt Template

This prompt is designed for evaluation engineers and AI quality teams who need a repeatable, rubric-driven method for scoring the faithfulness of a generated answer against a set of provided evidence passages. It applies a multi-level fidelity scale (fully faithful, partially faithful, unfaithful, contradictory) and returns a structured score with detailed reasoning. Use this prompt as an LLM-judge in automated evaluation pipelines, regression test suites, or pre-release quality gates for RAG systems, summarization engines, and any generative workflow where hallucination carries product risk.

This prompt assumes you already have a generated answer and the exact evidence passages that were provided to the model. It does not retrieve evidence, rewrite answers, or handle user-facing refusal. It is a measurement instrument, not a correction tool. The ideal user is someone running batch evaluations, comparing model versions, or setting up CI/CD checks that block deployment when faithfulness scores drop below a defined threshold. You should not use this prompt when you need real-time guardrails on live traffic—pair it instead with a lightweight hallucination classifier for production gating, and reserve this rubric for offline evaluation and pre-release testing.

Before wiring this into a pipeline, ensure your evidence passages are clean, deduplicated, and exactly match what the generation model received. Mismatched evidence is the most common failure mode: if the judge sees different passages than the generator used, faithfulness scores become meaningless. Start by running this prompt against a golden dataset of 50–100 answer-evidence pairs with human-annotated faithfulness labels to calibrate your thresholds. Then integrate it into your eval harness with structured logging, score distribution tracking, and automatic alerts when 'contradictory' or 'unfaithful' rates spike above your risk tolerance.

This prompt template implements a multi-level faithfulness fidelity rubric designed for automated LLM-judge evaluation pipelines. It instructs the model to compare a generated answer against provided reference evidence and assign one of four fidelity levels: fully faithful, partially faithful, unfaithful, or contradictory. The template includes detailed scoring criteria for each level, calibration examples to anchor the judge's behavior, and a structured output schema suitable for regression testing and dashboard ingestion. Replace every square-bracket placeholder before sending this prompt to the model. This template is not intended for end-user interaction; it is a measurement instrument for quality teams and evaluation engineers.

textCopy
You are an evaluation judge assessing the faithfulness of a generated answer against provided reference evidence. Your task is to assign a fidelity score based on how well the answer is supported by the evidence alone. Do not use outside knowledge. Do not evaluate answer quality, style, or completeness beyond factual grounding.

## Fidelity Levels

**Fully Faithful (score: 3)**
- Every factual claim in the answer is directly supported by the evidence.
- No claim contradicts or extends beyond the evidence.
- The answer may omit details present in the evidence without penalty.

**Partially Faithful (score: 2)**
- Most claims are supported, but one or more claims are:
  - Not directly stated in the evidence (inference without grounding), OR
  - Slightly exaggerated or overgeneralized beyond what the evidence supports.
- No claims directly contradict the evidence.

**Unfaithful (score: 1)**
- One or more claims are unsupported by the evidence and represent fabrication, hallucination, or substantial overstatement.
- The answer introduces facts, figures, or relationships not present in the evidence.
- No claims directly contradict the evidence, but significant portions are ungrounded.

**Contradictory (score: 0)**
- One or more claims directly contradict the evidence.
- The answer states the opposite of what the evidence says, reverses a relationship, or asserts a fact that the evidence explicitly refutes.

## Calibration Examples

[EXAMPLES]

## Input

**Evidence:**
[EVIDENCE]

**Answer to Evaluate:**
[ANSWER]

## Output Instructions

Return a JSON object with this exact schema:

[OUTPUT_SCHEMA]

## Constraints

[CONSTRAINTS]

Adaptation guidance: Replace [EVIDENCE] with the reference text passages the answer should be grounded in. Replace [ANSWER] with the generated answer under evaluation. Populate [EXAMPLES] with at least two calibration examples per fidelity level, each showing an evidence-answer pair and the correct score with a brief justification. Define [OUTPUT_SCHEMA] as a JSON schema including fields for fidelity_score (integer 0-3), fidelity_level (string matching one of the four levels), justification (string explaining the score), and unsupported_claims (array of strings identifying specific ungrounded or contradictory claims). Use [CONSTRAINTS] to specify additional rules such as "if the evidence is empty, score 1 (unfaithful) for any non-empty answer" or "treat hedging language as a claim if it asserts a factual likelihood." Test the populated template against a golden dataset of scored evidence-answer pairs before deploying to production evaluation pipelines. For high-stakes domains, always include human review of a sample of judge outputs to detect calibration drift.

The Faithfulness Fidelity Score Rubric prompt is designed to operate as an LLM judge inside an automated evaluation harness, not as a one-off manual check. The typical integration pattern places this prompt after answer generation and before the response reaches a user or a downstream system. You send the generated answer and the source evidence set as [ANSWER] and [EVIDENCE] inputs, receive a structured fidelity score with reasoning, and route the output based on the score. For production pipelines, treat this judge prompt as a gate: scores of fully_faithful pass through, partially_faithful triggers a warning log and optional re-generation, unfaithful blocks the answer and escalates for review, and contradictory immediately halts the pipeline and flags the output for human investigation. The rubric's four-level scale maps cleanly to automated decision thresholds that operations teams can monitor and tune over time.

Validation must happen in the application layer, not inside the prompt. After the model returns a response, parse the JSON output and confirm that the fidelity_score field contains exactly one of the four allowed enum values: fully_faithful, partially_faithful, unfaithful, or contradictory. Reject any response where the score is missing, misspelled, or accompanied by malformed JSON. Implement a retry loop with a maximum of two additional attempts, appending the validation error message to the next request so the model can self-correct. If the third attempt still fails validation, log the raw output, record the failure in your eval database, and escalate to a human reviewer rather than silently defaulting to a safe score. For high-stakes domains such as healthcare or legal review, add a second judge pass using a different model or a rotated prompt variant to reduce single-model bias, and require agreement between judges before accepting a fully_faithful verdict.

Logging and observability are essential because faithfulness scores drift as models, evidence sources, and user queries change. Capture the full input pair, the returned score, the reasoning text, the model version, the prompt template version, and a timestamp for every evaluation. Store these records in a structured format that your monitoring dashboard can query. Set up alerts for score distribution shifts: if the rate of unfaithful or contradictory scores suddenly increases, your retrieval pipeline or generation model may have degraded. For regression testing, maintain a golden dataset of answer-evidence pairs with known faithfulness labels, run this rubric prompt against the dataset on every prompt or model change, and block the release if the F1 score on fully_faithful detection drops below a predefined threshold. The rubric's calibration examples inside the prompt serve as anchor points, but external eval datasets provide the ground truth you need for release decisions.

Model choice matters for judge reliability. Prefer models with strong instruction-following and structured output capabilities for the judge role. If you are evaluating outputs from a smaller or faster model, the judge model should be at least as capable, and ideally more capable, than the generator. For cost-sensitive pipelines, batch evaluations during offline testing with a strong judge model, then use a lighter-weight classifier prompt for real-time gating in production. When the evidence set is large, truncate or chunk it before passing it to the judge, but always include the specific passages the generator cited. If the generator did not provide citations, run an alignment prompt first to map answer claims to evidence spans, then feed both the answer and the aligned evidence into this rubric prompt. Never ask the judge to search for supporting evidence inside a massive document dump; its reliability degrades sharply when evidence is buried in noise.

Human review integration is the final safety layer. Route all contradictory scores and a random sample of unfaithful scores to a review queue where a human verifies the judge's verdict. Use these human judgments to measure judge accuracy over time and to identify prompt weaknesses. If human reviewers consistently overturn the judge on specific score categories, update the rubric's calibration examples or adjust the score definitions. The goal is not to remove humans from the loop but to make their review time more valuable by pre-sorting outputs so they focus on the cases where the model is least reliable. Wire the human verdict back into your eval database as a training signal for future rubric improvements and as an audit trail for compliance reviews.