Semantic Equivalence Evaluation Prompt Template

Use this prompt when you need to verify that a change to your system prompt, model version, or generation parameters has not altered the core meaning of your AI's outputs. The primary job-to-be-done is regression testing for semantic drift: you have a 'golden' set of inputs and their corresponding reference outputs from a known-good version of your system, and you need to confirm that a new version produces outputs that convey the same information, intent, and constraints, even if the wording is different. The ideal user is an AI engineer or technical lead responsible for a prompt release pipeline, where every change must be proven safe against a curated test suite before deployment.

This prompt is designed for high-signal, structured comparison. It is not a general-purpose quality scorer, a chatbot preference test, or a replacement for human evaluation of subjective qualities like 'tone' or 'style.' It works best when the expected behavior is well-defined and the reference output is a concrete example of a correct response. You should provide the model with clear grading criteria, such as 'must contain the same key entities and their relationships' or 'must not introduce new constraints not present in the reference.' The prompt is most effective when the outputs being compared are relatively concise and self-contained, such as API responses, extracted data, or single-turn answers, rather than long-form creative prose.

Do not use this prompt when the acceptable output space is extremely broad or when 'equivalence' is inherently subjective without a detailed rubric. For example, comparing two valid but stylistically different marketing slogans requires preference judgment, not semantic equivalence. Similarly, avoid using this prompt for safety-critical decisions without a human-in-the-loop review step. The model's judgment of equivalence is a strong signal, but it can miss subtle factual errors or omissions that a domain expert would catch. Always pair this prompt with a pass/fail threshold and a spot-check process for borderline cases. For a complete regression testing workflow, combine this prompt with the Regression Test Suite Runner and Prompt Version Comparison Evaluation playbooks.

This template is the core instruction block for an LLM judge that determines whether two outputs convey the same meaning, intent, and constraints even when wording, structure, or examples differ. Use it when you need to validate that a prompt change, model migration, or retrieval update hasn't altered the substance of what your system produces. The template is designed to be dropped into an evaluation harness that supplies the reference output, candidate output, and any domain-specific equivalence criteria.

textCopy
You are an evaluation judge assessing semantic equivalence between two AI-generated outputs.

Your task: Determine whether OUTPUT_A and OUTPUT_B convey the same information, intent, constraints, and conclusions, even if wording, structure, examples, or formatting differ.

OUTPUT_A (reference):

[REFERENCE_OUTPUT]

codeCopy

OUTPUT_B (candidate):

[CANDIDATE_OUTPUT]

codeCopy

Evaluation context: [EVALUATION_CONTEXT]

Domain-specific equivalence criteria:
[DOMAIN_CRITERIA]

Equivalence dimensions to assess:
1. Factual content: Do both outputs contain the same core facts, data points, and claims?
2. Intent and purpose: Do both outputs serve the same user need and accomplish the same goal?
3. Constraints and limitations: Do both outputs respect the same boundaries, disclaimers, and scope?
4. Conclusions and recommendations: Do both outputs reach the same conclusions or provide equivalent guidance?
5. Omissions and additions: Does either output omit critical information present in the other, or add unsupported claims?

Scoring scale:
- EQUIVALENT: Same meaning across all dimensions; differences are purely stylistic or structural.
- SUBSTANTIALLY_EQUIVALENT: Same core meaning; minor differences in detail, emphasis, or phrasing that don't change the outcome.
- PARTIALLY_EQUIVALENT: Overlapping meaning but one output contains material additions, omissions, or shifts in emphasis that could affect user decisions.
- NOT_EQUIVALENT: Different facts, contradictory conclusions, missing critical information, or changed intent.

Output format (JSON only):
{
  "equivalence_rating": "EQUIVALENT | SUBSTANTIALLY_EQUIVALENT | PARTIALLY_EQUIVALENT | NOT_EQUIVALENT",
  "confidence": 0.0-1.0,
  "dimension_scores": {
    "factual_content": "equivalent | minor_difference | material_difference",
    "intent_and_purpose": "equivalent | minor_difference | material_difference",
    "constraints_and_limitations": "equivalent | minor_difference | material_difference",
    "conclusions_and_recommendations": "equivalent | minor_difference | material_difference",
    "omissions_and_additions": "none | minor | material"
  },
  "differences_found": [
    {
      "dimension": "string",
      "description": "Specific difference identified",
      "severity": "minor | material",
      "output_a_excerpt": "relevant excerpt or null",
      "output_b_excerpt": "relevant excerpt or null"
    }
  ],
  "rationale": "Brief explanation of the overall rating"
}

[ADDITIONAL_CONSTRAINTS]

Adapt this template by filling the square-bracket placeholders with your specific context. [REFERENCE_OUTPUT] should contain the known-good output from your baseline prompt or golden dataset. [CANDIDATE_OUTPUT] is the new output you're evaluating. [EVALUATION_CONTEXT] describes the task the outputs were responding to—include the original user input, system instructions, or retrieval context so the judge understands what both outputs were trying to accomplish. [DOMAIN_CRITERIA] is where you specify what equivalence means for your use case: for a medical summary, factual accuracy might dominate; for a creative brief, tone and intent alignment might matter more. [ADDITIONAL_CONSTRAINTS] lets you inject output format rules, word limits, or domain-specific instructions that the judge should consider when comparing outputs. For high-stakes domains, always pair this template with human calibration: run a sample of judge decisions past a domain expert and measure inter-rater agreement before trusting automated equivalence scores in your release pipeline.

The semantic equivalence prompt is rarely run in isolation. It belongs inside an evaluation harness that feeds it pairs of outputs, collects judgments, and routes uncertain cases for human review. The harness should treat the LLM judge as a fallible scorer, not an oracle. Every judgment must carry a confidence signal, and the system must know when to escalate. Wire the prompt into a pipeline that accepts a batch of output pairs—typically from a regression test suite or a prompt version comparison run—and returns structured equivalence decisions with supporting evidence.

Build the harness around a validation layer that enforces the output schema before trusting the result. The prompt should return a JSON object with at minimum equivalent (boolean), confidence (0.0–1.0), and rationale (string). If the model returns malformed JSON, missing fields, or a confidence value below your threshold (commonly 0.7–0.8), route the pair to a human review queue. For high-stakes domains such as healthcare or legal, require human sign-off on all equivalent: false judgments and any judgment with confidence below 0.9. Log every judgment with the prompt version, model, timestamp, and raw output for auditability. Use structured logging that captures the input pair hash, the judge's decision, and the final disposition (auto-accepted, auto-rejected, escalated, human-overridden). This log becomes your calibration dataset for measuring judge alignment over time.

Implement retry logic with care. If the model returns invalid JSON, retry once with a stricter format reminder appended to the prompt. If the second attempt fails, escalate to human review rather than retrying indefinitely. For non-deterministic judgments—where repeated runs on the same pair produce different equivalence decisions—flag the test case as flaky and exclude it from automated gating until the evaluation criteria are tightened or the prompt is stabilized. Choose a model with strong instruction-following for the judge role; GPT-4, Claude 3.5 Sonnet, and Gemini 1.5 Pro are common choices. Avoid using the same model family for both generation and judging when independence matters. Run calibration batches monthly by sampling 50–100 judged pairs and comparing LLM decisions against human raters. Track Cohen's kappa or raw agreement rate, and trigger a judge prompt review if agreement drops below 0.8.

When integrating with a CI/CD pipeline, treat the equivalence judge as a regression gate. After every prompt change, run the full golden dataset through both the old and new prompt versions, then feed every output pair through the equivalence judge. Aggregate results into a pass/fail report: a pair is a regression if equivalent is false and the new output is worse. Block deployment if regressions exceed a threshold (commonly zero for critical paths, or a small percentage with documented justifications). Store the judge's rationale for every flagged pair so the team can triage without re-running the evaluation. Avoid using the equivalence judge as the sole gate for safety-critical outputs—always pair it with domain-specific validators, schema checks, and human review for high-risk categories.