LLM Judge Rubric for Quote Fidelity

This prompt is for evaluation engineers and AI system builders who need to automate the quality assessment of a quote-verification pipeline. The core job-to-be-done is to produce a calibrated, structured rubric that an LLM judge can use to grade the output of another AI system—specifically, how well that system verified a quote against its source material. The ideal user is someone building a production evaluation harness, not a one-off manual reviewer. They need a rubric that defines clear fidelity dimensions, provides concrete scoring anchors, and includes instructions for pairwise comparison against human-annotated ground truth. The required context is a set of human-labeled examples that establish what 'correct' quote verification looks like, including edge cases like context-stripping, near-miss paraphrases, and benign truncation.

Do not use this prompt when you need a direct quote-to-source comparison for editorial review. That workflow is covered by the 'Quote-to-Source Comparison Prompt Template' in this pillar. This rubric prompt is a meta-evaluation tool—it grades the grader, not the content. It is also inappropriate for low-stakes, single-shot verification tasks where a simple binary check suffices. The rubric is designed for calibration against a golden dataset, which means you must have human-annotated ground truth before you begin. Without that, the rubric's scoring anchors will float and inter-rater reliability tracking becomes meaningless. The output is not a verification report; it is a machine-readable evaluation schema that feeds into your CI/CD pipeline for prompt regression testing.

Before using this prompt, ensure you have a stable set of human-annotated quote-verification examples that capture the failure modes you care about: misquotation, context-stripping, meaning distortion, and fabrication. The rubric's value scales with the quality of your ground truth. If your human annotators disagree on what constitutes a fidelity failure, the LLM judge will amplify that noise. Start with a small calibration set, run this prompt to generate a draft rubric, then iterate with your human reviewers until the scoring anchors produce consistent agreement. The next step is to wire this rubric into an evaluation harness that tracks inter-rater reliability between the LLM judge and your human annotators over time.

This template is designed to act as an LLM judge for evaluating the quality of an automated quote-verification system. It compares a system's verification output against a human-annotated ground truth label and explanation. The core job is to produce a structured, dimension-level score and a final fidelity judgment, enabling consistent, automated evaluation across a large dataset. Use this when you need to track the performance of your verification pipeline over time, compare different prompt versions, or establish a quality gate before deployment.

textCopy
You are an expert evaluation judge for quote-verification systems. Your task is to grade a system's verification output against a human-annotated ground truth for a single quote-source pair. You must be strict, consistent, and evidence-based.

# INPUT DATA
[QUOTE]
[SOURCE_TEXT]
[SYSTEM_VERIFICATION_OUTPUT]
[HUMAN_GROUND_TRUTH]

# RUBRIC DIMENSIONS
You will grade the system's output on the following dimensions, each on a 1-5 scale:

1.  **Fidelity Score (1-5):** How accurately does the system's final verification status (e.g., 'Accurate', 'Misquoted', 'Unverifiable') match the human ground truth status?
    *   5: Perfect match.
    *   3: Partially correct (e.g., correct status but wrong confidence, or correct flag but wrong severity).
    *   1: Completely incorrect status.

2.  **Evidence Grounding (1-5):** How well does the system's provided evidence and rationale support its conclusion, and how well does it align with the evidence cited in the human ground truth?
    *   5: Evidence is precise, correctly cited, and directly supports the conclusion, matching the human annotator's key evidence points.
    *   3: Evidence is generally relevant but may be vague, miscited, or misses a key piece of evidence from the ground truth.
    *   1: Evidence is irrelevant, hallucinated, or contradicts the conclusion.

3.  **Discrepancy Explanation (1-5):** If a discrepancy exists, how clearly and accurately does the system explain the difference between the quote and the source? If no discrepancy exists, does it correctly avoid inventing one?
    *   5: Explanation is precise, specific, and correctly identifies the nature and location of the discrepancy (or correctly states there is none).
    *   3: Explanation is vague, partially incorrect, or misses a key nuance of the discrepancy.
    *   1: Explanation is factually wrong, hallucinates a non-existent discrepancy, or fails to identify a clear one.

# CONSTRAINTS
- Base your evaluation ONLY on the provided [QUOTE], [SOURCE_TEXT], [SYSTEM_VERIFICATION_OUTPUT], and [HUMAN_GROUND_TRUTH].
- Do not use outside knowledge.
- If the system output is malformed or empty, assign a score of 1 for all dimensions and explain why in the rationale.

# OUTPUT FORMAT
You must output a single, valid JSON object with the following schema:
{
  "evaluation_id": "[UNIQUE_ID]",
  "scores": {
    "fidelity_score": {
      "score": int,
      "rationale": "A concise, evidence-based explanation for the score."
    },
    "evidence_grounding": {
      "score": int,
      "rationale": "A concise, evidence-based explanation for the score."
    },
    "discrepancy_explanation": {
      "score": int,
      "rationale": "A concise, evidence-based explanation for the score."
    }
  },
  "overall_judgment": "PASS" | "FAIL",
  "overall_rationale": "A summary explaining the overall judgment, referencing the dimension scores."
}

# PASS/FAIL CRITERIA
- A "PASS" requires a score of 4 or higher on ALL dimensions.
- Any score of 3 or below results in a "FAIL".

To adapt this template, replace the placeholders with your actual data. The [UNIQUE_ID] in the output schema should be generated dynamically by your harness to track individual evaluations. The strict PASS/FAIL criteria are designed for a high-recall quality gate; you can adjust the threshold (e.g., to 3 or higher) for a development environment. For high-stakes domains like legal or medical quote verification, consider adding a fourth dimension for 'Source Context Adherence' and requiring human review for all FAIL judgments before accepting them as true negatives.

This rubric prompt is designed to operate as a scoring function inside an evaluation harness, not as a one-off chat interaction. The typical integration pattern is: a quote-verification system produces an output (a fidelity score, a set of flagged discrepancies, or a structured alignment report), and this rubric prompt grades that output against a human-annotated ground-truth label. The harness calls the rubric prompt once per evaluation sample, collects the structured score and rationale, and aggregates results across the full test set. This architecture separates the system-under-test from the judge, preventing self-evaluation bias and enabling independent calibration of the judge against human annotators.

Validation and output contract enforcement is the first integration concern. The rubric prompt must return a strict JSON schema containing at minimum: fidelity_score (a numeric value on the defined scale), rationale (a free-text explanation tied to specific rubric dimensions), and dimension_scores (a dictionary of per-dimension scores matching the rubric's axes, such as wording_accuracy, context_preservation, and intent_alignment). Implement a post-processing validator that checks schema compliance, score range adherence, and the presence of all required dimensions. On validation failure, retry once with the validation error message appended to the prompt as a correction instruction. If the retry also fails, log the sample for human review and exclude it from aggregate metrics rather than silently accepting a malformed score. This is especially important when the rubric is used as a release gate for quote-verification model updates.

Inter-rater reliability tracking should be built into the harness from day one. For each evaluation batch, include a subset of samples that were also scored by at least two human annotators. The harness computes agreement metrics (Cohen's kappa or weighted kappa for ordinal rubric scales, Krippendorff's alpha for multi-annotator setups) between the LLM judge and each human, and between human annotators as a baseline. Store these metrics per evaluation run so you can detect judge drift over time. If the LLM-human agreement drops below a pre-defined threshold (e.g., kappa < 0.7), the harness should flag the run for investigation rather than silently accepting degraded evaluation quality. This is not a one-time calibration step; it is a continuous monitoring requirement for any production evaluation pipeline.

Model choice and temperature directly affect rubric reliability. Use a model with strong instruction-following and reasoning capabilities (GPT-4, Claude 3.5 Sonnet, or equivalent) with temperature set to 0 for deterministic scoring. Avoid using the same model family for both the system-under-test and the judge when possible; cross-model evaluation reduces correlated error modes. For cost-sensitive pipelines running large evaluation batches, consider routing a random 20% of samples to a stronger judge model and using that subset to calibrate a faster, cheaper judge on the remaining 80%. Log which model scored each sample so you can disaggregate metrics by judge model during analysis.

Pairwise comparison mode requires a different harness structure than pointwise scoring. When the rubric is used to compare two quote-verification outputs (e.g., old system vs. new system on the same input), the harness must present both outputs to the judge in randomized order with position labels anonymized. The judge returns a preference (output_a_better, output_b_better, or tie) along with a dimension-by-dimension comparison rationale. The harness then unblinds the positions and computes win rates. For statistical significance, run at least 100 pairwise comparisons before drawing conclusions. Avoid presenting the same pair twice with reversed positions in the same batch, as this can create anchoring effects that distort the judge's behavior.

Failure modes to instrument include: score inflation on ambiguous quotes where the ground truth is genuinely uncertain (the judge should use the middle of the scale, not default to high scores); dimension collapse where the judge stops distinguishing between rubric dimensions and assigns identical scores across all axes (detect this by monitoring per-dimension variance per sample); and position bias in pairwise mode where the judge systematically prefers the first or second output regardless of quality (detect by tracking win rates by position and randomizing order). Build dashboards or alerting rules for each of these failure modes. When a failure mode is detected, do not silently discard the data—quarantine the affected evaluation run and investigate whether the rubric prompt, the judge model, or the ground-truth labels need revision.