Head-to-Head Output Comparison Prompt Template

This prompt is designed for evaluation engineers and ML platform teams who need to make a forced-choice preference decision between two model outputs. The core job-to-be-done is producing a structured, justifiable verdict that answers: 'Which output is better for this specific request, and why?' It is the atomic unit of work in RLHF data pipelines, champion-challenger model testing, and prompt version regression suites where relative quality matters more than an absolute score. The ideal user is someone wiring this into an automated evaluation harness—not a product manager spot-checking outputs by hand.

Use this prompt when you have exactly two outputs to compare against a shared input and a clear set of evaluation dimensions. It works best when the comparison criteria are stable (e.g., accuracy, safety, instruction adherence) and you need a repeatable verdict with a confidence score and key differentiators. Do not use this prompt for absolute quality scoring of a single output, for ranking more than two candidates, or for evaluating multi-turn conversations without adapting the template to include turn history. It is also the wrong tool when the two outputs are not comparable—for example, when they answer different interpretations of an ambiguous prompt, or when one output refused and the other attempted an answer. In those cases, a rubric-based absolute evaluation or a tie-breaking prompt with explicit criteria will serve you better.

Before deploying this prompt into a production pipeline, you must implement position bias mitigation. The order of Output A and Output B in the prompt influences judge behavior, so your harness must randomize presentation order and track which position each output occupied. You should also run inter-rater consistency checks by comparing multiple judge calls on the same pair, and calibrate confidence scores against human preference labels if this prompt feeds a training dataset. The next section provides the copy-ready template with placeholders for your inputs, criteria, and output schema.

This prompt template is designed for evaluation engineers who need to compare two model outputs side by side and produce a structured preference verdict. The template includes placeholders for the two outputs, evaluation criteria, position randomization instructions, and a required output schema. Use this template when you need relative quality judgments rather than absolute scores, such as in RLHF data pipelines, champion-challenger testing, or model selection workflows.

textCopy
You are an expert output evaluator. Your task is to compare two AI-generated outputs and determine which one is better according to the specified criteria.

## Evaluation Criteria
[CRITERIA]

## Output A
[OUTPUT_A]

## Output B
[OUTPUT_B]

## Instructions
1. Read both outputs carefully against the evaluation criteria above.
2. Ignore the order in which the outputs are presented. Output A and Output B are randomly ordered. Do not assume Output A is the first response or the preferred response.
3. Identify the key differences between the two outputs.
4. Select your preference: "A", "B", or "TIE".
5. Provide a structured justification for your choice.
6. Assign a confidence score from 0.0 to 1.0 reflecting how certain you are in your preference.

## Constraints
[CONSTRAINTS]

## Output Schema
Return your evaluation as a JSON object with the following structure:
{
  "preference": "A" | "B" | "TIE",
  "confidence": 0.0-1.0,
  "justification": "Detailed explanation of why the preferred output is better, referencing specific criteria.",
  "key_differentiators": [
    "Specific difference 1",
    "Specific difference 2"
  ],
  "criteria_breakdown": {
    "[CRITERION_NAME_1]": {"winner": "A" | "B" | "TIE", "explanation": "..."},
    "[CRITERION_NAME_2]": {"winner": "A" | "B" | "TIE", "explanation": "..."}
  },
  "tie_decision": null | {"reason": "Why no clear winner", "tie_breaker_applied": "Which secondary criterion was used if any"}
}

## Examples
[EXAMPLES]

To adapt this template, replace the square-bracket placeholders with your specific evaluation context. The [CRITERIA] placeholder should contain your evaluation dimensions such as accuracy, clarity, completeness, safety, or instruction adherence. The [CONSTRAINTS] placeholder can include word limits, format requirements, or domain-specific rules. The [EXAMPLES] placeholder should contain one or two few-shot examples showing the desired output format with realistic comparisons. For high-stakes evaluations, add a [RISK_LEVEL] field and require human review when confidence falls below a threshold. Always randomize the order of Output A and Output B in your application code before inserting them into this template to mitigate position bias.

The head-to-head comparison prompt is rarely used in isolation. In production, it sits inside a harness that randomizes output order, validates the judge's response structure, retries on malformed outputs, and logs every decision for later calibration analysis. The harness is responsible for fairness, not the prompt. The prompt assumes it receives two outputs labeled generically; the harness must enforce that labeling and strip any model-identifying metadata before the prompt is assembled.

Start by randomizing the presentation order of Output A and Output B for every comparison request. Store the original order in your logging payload so you can detect position bias during retrospective analysis. If your evaluation platform supports it, run each comparison twice with swapped positions and flag cases where the preference flips—these are candidates for tie-breaking or human review. For model choice, use a capable judge model with strong instruction-following behavior (e.g., Claude 3.5 Sonnet, GPT-4o) and keep the judge model version fixed across an evaluation campaign to avoid judge drift. Set temperature=0 or a very low value to maximize output determinism and schema compliance.

Validate the judge's JSON output against a strict schema before accepting the verdict. Required fields include preference (must be one of Output A, Output B, or Tie), confidence (a float between 0 and 1), justification (a non-empty string), and key_differentiators (an array of at least one string). If validation fails, retry up to two times with the same prompt, appending the validation error message as additional context. After two failed retries, log the failure and route the comparison for human review. For high-stakes evaluation pipelines, sample 5-10% of judge decisions for human audit and track inter-rater agreement over time. When agreement drops below a threshold (e.g., Cohen's kappa < 0.7), pause automated evaluation and investigate for judge drift, prompt degradation, or changing data distributions.

Wire the harness to emit structured logs containing the randomized order, the judge's raw response, the validated verdict, retry counts, and the human audit outcome if applicable. Store these logs in your evaluation database alongside the prompt version hash and judge model identifier. This audit trail is essential for calibrating confidence scores against human agreement rates and for defending evaluation results during model release reviews. Avoid treating the judge's confidence score as a calibrated probability until you have measured it against human preference labels on your specific data distribution.