Prompt Version Comparison Evaluation Prompt

This playbook is for prompt engineering teams that need a data-driven decision process for promoting a prompt change. Instead of relying on ad-hoc spot checks or a single aggregate score, this prompt runs a candidate version and a baseline version against a shared test suite and produces per-example win/loss/tie judgments with explanations. The output is a decision-support report that includes aggregate scoring, confidence intervals, and a clear recommendation. Use it when you have a candidate prompt, a baseline prompt, and a set of test cases with expected outputs.

The prompt requires three concrete inputs to function: a [BASELINE_PROMPT] that represents the current production version, a [CANDIDATE_PROMPT] that is the proposed change, and a [TEST_SUITE] containing input-output pairs with expected behavior. Each test case should include the input, the expected output, and optional tolerance thresholds for semantic equivalence. The prompt works best when your test suite covers a representative distribution of production traffic, including edge cases, boundary conditions, and known failure modes. If your test suite is sparse or unrepresentative, the comparison results will be unreliable regardless of how well the prompt is constructed.

Do not use this prompt when you lack a defined test suite, when the expected outputs are purely subjective and cannot be evaluated programmatically, or when the prompt change is cosmetic and does not affect functional behavior. For subjective quality dimensions like tone or style, pair this prompt with a separate LLM judge evaluation using calibrated rubrics. For cosmetic changes like whitespace or comment adjustments, skip the comparison entirely and rely on your version control diff. The next section provides the copy-ready prompt template you can adapt for your own evaluation harness.

This prompt is the core instruction set for a Prompt Version Comparison Evaluation. It is designed to be pasted into your evaluation harness, where it will receive two prompt versions and a set of test cases. The model acts as a judge, comparing outputs side-by-side and producing structured win/loss/tie judgments with detailed explanations. Before using this template, ensure you have a validated golden test set and a clear definition of what constitutes a better output for your specific use case.

textCopy
You are an expert prompt evaluator. Your task is to compare two prompt versions against a shared test suite and determine which version produces better outputs.

## INPUT DATA
- Test Suite: [TEST_SUITE]
  A set of test cases, each containing an [INPUT] and an [EXPECTED_OUTPUT_CONTRACT].
- Prompt Version A: [PROMPT_VERSION_A]
- Prompt Version B: [PROMPT_VERSION_B]
- Outputs for Version A: [OUTPUTS_A]
  The model's responses to each test case using Prompt Version A.
- Outputs for Version B: [OUTPUTS_B]
  The model's responses to each test case using Prompt Version B.

## EVALUATION CRITERIA
For each test case, compare the two outputs against the [EXPECTED_OUTPUT_CONTRACT] and determine a winner based on these dimensions:
- **Accuracy**: Which output better matches the expected facts, data, or logical conclusions?
- **Schema Compliance**: Which output better adheres to the required format, field types, and structure?
- **Completeness**: Which output addresses all required elements without missing critical information?
- **Constraint Adherence**: Which output better respects explicit constraints like tone, length, or prohibitions?
- **Safety and Refusal Correctness**: If refusal is expected, which output refuses more appropriately? If refusal is incorrect, penalize accordingly.

## OUTPUT FORMAT
Return a single JSON object with the following structure:
{
  "aggregate_results": {
    "version_a_wins": <integer>,
    "version_b_wins": <integer>,
    "ties": <integer>,
    "total_test_cases": <integer>,
    "version_a_win_rate": <float between 0 and 1>,
    "version_b_win_rate": <float between 0 and 1>,
    "confidence_interval_95": "<string describing the 95% confidence interval for the difference>"
  },
  "per_example_results": [
    {
      "test_case_id": "<string>",
      "judgment": "<version_a_wins | version_b_wins | tie>",
      "explanation": "<concise reason for the judgment, referencing specific criteria>",
      "key_differences": "<summary of what differed between the two outputs>"
    }
  ],
  "overall_assessment": "<narrative summary of the comparison, including which version is recommended and any notable patterns or risks>"
}

## CONSTRAINTS
- Do not hallucinate differences. Only report differences you can observe in the provided outputs.
- If both outputs are equally good or equally flawed, mark as a tie and explain why.
- If a test case is invalid or cannot be evaluated, mark it as a tie and note the issue in the explanation.
- Base your judgment strictly on the provided [EXPECTED_OUTPUT_CONTRACT], not on personal preference.

To adapt this template, replace the square-bracket placeholders with your actual data. The [TEST_SUITE] should be a structured list of test cases, each with a unique ID, an input, and an expected output contract. The [EXPECTED_OUTPUT_CONTRACT] can be a full golden output, a schema definition, or a set of pass/fail rules. The [OUTPUTS_A] and [OUTPUTS_B] should be pre-generated by running each prompt version against the test suite before invoking this evaluation prompt. For high-stakes decisions, always have a human review the per-example explanations and the overall assessment before promoting a prompt change.

This prompt is designed to be the core comparison engine in an automated prompt evaluation harness, not a one-off manual review tool. It expects a structured test suite—a set of inputs paired with expected outputs or acceptance criteria—and two prompt versions to compare. The harness should iterate over each test case, call the model with this comparison prompt, and collect the per-example win/loss/tie judgments. The prompt's value comes from its ability to produce structured, machine-readable judgments that can be aggregated into a release gate report.

To wire this into an application, build a runner that loads your test suite from a version-controlled file (JSON Lines or a database), then for each test case: (1) execute both prompt versions against the input, capturing the raw outputs; (2) construct the comparison prompt by injecting the input, the two outputs, and your evaluation criteria into the [INPUT], [OUTPUT_A], [OUTPUT_B], and [EVALUATION_CRITERIA] placeholders; (3) call a capable model (GPT-4o, Claude 3.5 Sonnet, or equivalent) with response_format set to json_object and a strict JSON schema matching the expected output contract; (4) validate the returned JSON for required fields (judgment, winner, explanation, confidence), enum values (A, B, tie), and confidence score range (0.0–1.0). Failed validations should trigger a single retry with the validation error message appended to the prompt as additional context. Log every comparison result—including raw outputs, judgment, and validation status—to your observability platform for later trace analysis.

For high-stakes prompt changes where a bad promotion could degrade user-facing behavior, add a human review gate before aggregating results. After the harness completes all comparisons, compute aggregate metrics: win rate for each version, tie rate, and a 95% confidence interval for the win-rate difference using a binomial proportion test. If the confidence interval for the new version's improvement excludes zero and the win rate exceeds your pre-defined threshold (e.g., >55% with p<0.05), the harness can emit a go signal. Otherwise, it should emit no-go or review. Store the full comparison report—including per-example judgments, aggregate statistics, and the raw prompt versions—as an artifact in your CI/CD pipeline for auditability. Do not rely solely on this prompt's aggregate score; always spot-check a random sample of comparisons, especially ties and low-confidence judgments, to catch evaluator blind spots before promoting a prompt to production.