LLM Judge Prompt for Factual Accuracy Evaluation

The verified, correct answer against which the generated answer is compared.

The capital of France is Paris. It has been the capital since the 10th century.

Must be a non-empty string. Check length > 0. If ground truth is unknown or unavailable, this prompt is not applicable; use a source-grounded verification prompt instead.

The AI-generated answer that needs to be evaluated for factual accuracy.

Paris is the capital of France and was established as the capital in the year 987.

Must be a non-empty string. Check length > 0. If the generated answer is empty, the evaluation should return a score of 0 with a reason of 'no answer provided'.

A structured description of the scoring scale, definitions for each score level, and rules for handling partial correctness and extraneous information.

Score 1: Completely incorrect. Score 2: Partially correct with major errors. Score 3: Mostly correct with minor omissions. Score 4: Fully correct. Score 5: Fully correct and complete.

Must be a non-empty string. Validate that it contains a numeric scale definition. If criteria are missing, the judge may apply inconsistent standards. Consider storing this as a reusable template variable.

Specific factual dimensions the judge should prioritize, such as dates, names, quantities, or causal relationships.

dates, numerical quantities, proper nouns

Optional. If provided, must be a comma-separated string or null. If null, the judge evaluates all factual content equally. Validate format if not null.

Instructions for how to handle near-matches, paraphrases, and semantically equivalent statements that differ in wording.

Treat 'approximately 1000 years ago' as equivalent to 'since the 10th century' for date ranges.

Optional. If provided, must be a non-empty string. If null, the judge applies strict literal matching. Document the tolerance policy in your eval run metadata for reproducibility.

The exact schema the judge must return, including field names, types, and descriptions.

JSON object with fields: score (integer 1-5), justification (string), errors_found (array of strings), missing_facts (array of strings)

Must be a valid JSON Schema string or a structured description. Parse and validate the schema before sending. If the output format is invalid, the downstream parser will fail. Include a retry instruction in the harness if the judge output does not conform.

Calibrated examples of generated answers, ground truth answers, and the expected evaluation output to anchor the judge's scoring behavior.

Example 1: Generated='Paris is the capital of France.' GroundTruth='The capital of France is Paris.' Output: {score: 5, ...}

Optional but strongly recommended for score calibration. If provided, must be an array of objects with generated_answer, ground_truth_answer, and expected_output fields. Validate the array structure. If null, the prompt should still function but may exhibit score drift across runs.

integer (1-5)

Must be an integer between 1 and 5 inclusive. Reject if float, string, or out of range.

string

Must be non-empty and contain at least one explicit reference to a specific claim in [GENERATED_ANSWER].

array of objects

Array must not be empty. Each element must conform to the claim_verification schema.

string

Must be an exact text span extracted from [GENERATED_ANSWER]. Reject if claim text is not found verbatim in the answer.

string (enum)

Must be one of: 'fully_supported', 'partially_supported', 'contradicted', 'unsupported'. Reject any other value.

string or null

If verdict is 'fully_supported' or 'partially_supported', must contain a direct quote or specific passage reference from [GROUND_TRUTH]. If 'unsupported' or 'contradicted', must be null.

string

Must explain the reasoning linking the claim, evidence, and verdict. Reject if empty or purely generic.

string

Must summarize the distribution of verdicts and justify the final factual_accuracy_score. Reject if length is less than 20 characters.

All key entities, dates, quantities, and named concepts in [GENERATED_ANSWER] match [GROUND_TRUTH_REFERENCE] exactly or with semantically equivalent paraphrasing.

A named entity is replaced with a different entity, a date is off by more than one unit, or a quantity is materially incorrect.

Parse both texts for named entities and numeric values. Perform exact and fuzzy string matching. Flag any mismatch where the edit distance exceeds a defined threshold.

No claim in [GENERATED_ANSWER] introduces a specific fact not present in [GROUND_TRUTH_REFERENCE].

The answer adds a specific name, number, event, or causal claim absent from the reference. Vague stylistic additions are ignored.

Decompose [GENERATED_ANSWER] into atomic claims. For each claim, run a semantic entailment check against [GROUND_TRUTH_REFERENCE]. Flag any claim with entailment score below 0.9.

If [GENERATED_ANSWER] is partially correct, the score reflects the proportion of correct claims. Extraneous true facts from outside the reference are treated as neutral, not penalized.

A single critical error in a multi-claim answer causes a zero score. The judge penalizes correct supplementary information that was not requested.

Compute precision as (correct claims / total claims in answer). Use a human-annotated calibration set of 50 partial-correctness examples to validate the judge's scoring distribution.

Missing non-critical details (e.g., an adjective) does not reduce the score. Missing a core entity or event reduces the score proportionally.

The judge penalizes an answer for omitting a detail that was present in the reference but not strictly required by the question.

Create a dependency map of required facts from [GROUND_TRUTH_REFERENCE]. Score recall as (required facts present in answer / total required facts). Validate against a golden set of acceptable omission examples.

No statement in [GENERATED_ANSWER] logically contradicts [GROUND_TRUTH_REFERENCE].

The answer states the opposite of a reference fact (e.g., 'increased' vs 'decreased') or asserts a mutually exclusive condition.

Use a dedicated contradiction detection prompt as a secondary judge. Feed the contradicting statement and reference passage. Require a binary verdict with a confidence score above 0.95 to trigger a failure.

The judge ignores information in [GENERATED_ANSWER] that is true but outside the scope of [GROUND_TRUTH_REFERENCE]. It does not penalize harmless additions.

The judge deducts points for any statement not explicitly found in the reference, even if factually true.

Classify each claim as 'in-reference', 'out-of-reference-true', or 'out-of-reference-false'. Only the last category triggers a penalty. Validate this behavior with 20 examples containing known true extraneous facts.

Numerical values within a configurable tolerance (default ±5%) of the reference value are considered correct.

The judge requires exact string match for numbers, failing answers that round or approximate correctly.

Extract all numerical values from both texts. Apply a relative error calculation. Flag only pairs where |answer_value - reference_value| / |reference_value| > tolerance. Confirm tolerance with domain experts.

The judge's 1-5 score correlates with human expert ratings at Spearman's ρ ≥ 0.8 on a held-out calibration set.

The judge exhibits score inflation, giving high scores to partially incorrect answers, or score deflation, penalizing stylistic differences.

Run the judge against a calibration set of 100 answer-reference pairs with three independent human expert ratings. Calculate Spearman's rank correlation. Recalibrate the rubric or prompt instructions if correlation falls below 0.8.