Reference-Based Evaluation Prompt Template

This prompt is for AI quality engineers and platform teams who need to compare a generated output against a known golden reference. It produces a similarity score with semantic equivalence detection, acceptable variation boundaries, and exact-match versus paraphrase differentiation. Use this when you have a curated golden dataset and need automated, repeatable evaluation that goes beyond string matching. This prompt belongs in regression test suites, CI/CD eval gates, and any workflow where you must quantify how closely a new prompt version reproduces expected behavior.

The ideal user has already built a golden dataset of input-output pairs that represent correct, expected behavior. You should wire this prompt into an automated harness that iterates over test cases, calls the prompt for each pair, and aggregates the similarity scores into a pass/fail report. The prompt handles both exact-match verification and semantic equivalence, which means it can distinguish between a permissible paraphrase ('The user's balance is $42.50') and an unacceptable deviation ('The balance appears sufficient'). Configure the acceptable variation boundaries in the [CONSTRAINTS] placeholder to match your domain's tolerance. For high-stakes domains like healthcare or finance, set strict boundaries and always include a human review step for borderline scores.

Do not use this prompt for open-ended generation tasks where no reference exists; use the Reference-Free Evaluation Prompt Template instead. Do not use this for pairwise A/B comparisons; use the Pairwise Comparison Judge Prompt Template. Avoid this prompt when your golden dataset is small or unrepresentative, as the scores will give false confidence. If you observe score inflation over time, audit your golden dataset for memorization leakage or drift. The next section provides the copy-ready template you can adapt and integrate into your eval harness.

This template is the core instruction set you will send to an LLM judge. It is designed to be wired into an automated evaluation harness where the [CANDIDATE_OUTPUT] is the text your system produced and the [REFERENCE_OUTPUT] is the known-good, human-validated answer. The prompt forces the model to act as a strict evaluator, distinguishing between exact matches, acceptable paraphrases, and substantive errors, rather than giving a vague 'looks good' rating.

textCopy
You are an expert output evaluator. Your task is to compare the [CANDIDATE_OUTPUT] against the [REFERENCE_OUTPUT] and produce a structured similarity assessment.

## Evaluation Criteria
1. **Exact Match**: The candidate is character-for-character identical to the reference, ignoring leading/trailing whitespace.
2. **Semantic Equivalence**: The candidate conveys the same core facts, intent, and meaning as the reference, even if wording, structure, or examples differ. This is a pass.
3. **Acceptable Variation**: The candidate adds minor, non-contradictory details or omits non-essential context without changing the core answer. This is a partial pass.
4. **Substantive Difference**: The candidate contradicts a fact in the reference, introduces a hallucination, changes a key decision, or omits a critical constraint. This is a fail.

## Constraints
- [CONSTRAINTS]

## Output Schema
Respond ONLY with a valid JSON object matching this schema:
{
  "similarity_score": <float 0.0-1.0>,
  "equivalence_class": "exact_match" | "semantic_equivalent" | "acceptable_variation" | "substantive_difference",
  "partial_credit": <float 0.0-1.0>,
  "rationale": "<concise explanation of the judgment>",
  "key_differences": ["<list of specific divergences>"]
}

## Partial Credit Rules
- exact_match: partial_credit = 1.0
- semantic_equivalent: partial_credit = 0.95
- acceptable_variation: partial_credit = 0.7
- substantive_difference: partial_credit = max(0.0, 1.0 - (number_of_critical_errors * 0.3))

## Inputs
[CANDIDATE_OUTPUT]:
"""
[CANDIDATE_OUTPUT]
"""

[REFERENCE_OUTPUT]:
"""
[REFERENCE_OUTPUT]
"""

## Examples
[EXAMPLES]

To adapt this template, start by populating the [CONSTRAINTS] placeholder with domain-specific rules, such as 'Ignore differences in markdown formatting' or 'A change in a numerical value is always a substantive difference.' The [EXAMPLES] placeholder should contain 2-3 few-shot demonstrations showing borderline cases, like a paraphrase that should be scored as semantic_equivalent versus one that drifts into acceptable_variation. If you are evaluating in a high-risk domain like healthcare or finance, add a constraint requiring the rationale to cite specific evidence from the reference, and always route substantive_difference results to a human review queue before accepting the score as final. Do not deploy this prompt without first running it against a calibration set of 20-30 known pairs to verify that the judge's similarity_score and equivalence_class assignments align with your team's expectations.

The reference-based evaluation prompt is not a one-off tool; it is a scoring function that should be embedded into a deterministic application harness. The harness is responsible for orchestrating the model call, validating the structured output, managing retries, and logging every judgment for auditability. Treat this prompt as a microservice with a strict contract: it receives a candidate output and a golden reference, and it must return a machine-readable similarity score with supporting evidence. The harness, not the prompt, enforces timeouts, handles malformed JSON, and decides whether a failing score blocks a release.

Implement the harness with these concrete components. Input validation: before calling the model, assert that both [CANDIDATE_OUTPUT] and [REFERENCE_OUTPUT] are non-empty strings and that any [ACCEPTABLE_VARIATION_RULES] are well-formed. Model selection: use a model with strong instruction-following and JSON mode support (e.g., gpt-4o, claude-3.5-sonnet). Avoid models prone to verbosity or format drift for this task. Structured output enforcement: call the model with response_format: { type: "json_object" } and provide the exact JSON schema from the prompt template. Post-processing: parse the response, validate it against the schema, and extract the similarity_score. If parsing fails, retry once with a repair prompt that includes the raw output and the schema. Retry logic: implement exponential backoff for rate limits (429) and server errors (5xx), but fail fast on persistent validation errors (after 2 retries) to avoid burning budget on a broken prompt. Logging: record the evaluation_id, prompt_version, model_id, candidate_output_hash, reference_output_hash, raw score, parsed justification, and latency for every call. This log becomes your eval audit trail.

For high-stakes workflows—such as gating a production deployment or evaluating safety-critical outputs—add a human review fallback. If the similarity score falls into a configured ambiguity band (e.g., 0.6–0.8), flag the evaluation for manual inspection rather than auto-passing or auto-failing. Do not treat the LLM judge as ground truth; treat it as a calibrated measurement instrument that requires ongoing monitoring. Regularly run the calibration prompt from the sibling playbook against a held-out set of human-labeled pairs to detect judge drift. Finally, integrate the harness into your CI/CD pipeline as a gating step: the deployment script calls the evaluation harness, checks the score against a [PASS_THRESHOLD], and blocks the release if the threshold is not met.