Blind A/B Output Evaluation Prompt

This prompt is for product and evaluation teams who need to compare two AI-generated outputs without the evaluator (human or LLM judge) knowing which model, prompt version, or system produced each one. The core job is to eliminate model identity bias—the tendency for raters to favor outputs from a known preferred model or to be influenced by stylistic fingerprints that reveal the source. Use this when you are running a champion-challenger test, selecting between two prompt variants, or building a preference dataset for RLHF where label quality depends on the evaluator judging content, not provenance.

The prompt is designed for controlled, pairwise comparison workflows where you can strip all identity signals before the outputs reach the judge. This means you must remove model names, version tags, confidence scores, and any system-level commentary that hints at the source. The prompt includes an evaluator blinding protocol that instructs the judge to treat both outputs as anonymous and a leakage detection check that asks the judge to flag any residual identity hints. Do not use this prompt when you need absolute quality scores against a fixed rubric—use a rubric-based scoring prompt instead. Do not use it when the comparison requires real-time tool execution, code validation, or external API calls that would reveal the model's capabilities asymmetrically.

Before deploying this prompt, verify that your output preprocessing pipeline truly strips identity signals. Common leakage vectors include markdown formatting differences, emoji usage patterns, refusal phrasing, and citation styles that correlate with specific models. Run a dry test where you present two outputs from the same model and confirm the judge does not systematically prefer one position. If position bias exceeds 5%, add position randomization to your harness. For high-stakes model selection decisions, always pair this prompt with human review on a sample of preference labels to calibrate the judge's alignment with your team's actual quality standards.

This prompt template is designed to be dropped directly into your evaluation harness. It forces the judge model to compare two outputs without knowing which model or prompt version produced them. The template uses square-bracket placeholders for all variable inputs—your application code should replace these before sending the request. The prompt includes explicit blinding instructions, position randomization requirements, and identity-leakage detection checks that prevent the evaluator from inferring model origin from stylistic tics, formatting patterns, or signature phrases.

textCopy
You are an impartial output evaluator. Your task is to compare two AI-generated outputs and select the better one based solely on the criteria below. You do not know and must not attempt to guess which model, prompt, or system produced either output. The outputs have been randomly assigned labels "Output A" and "Output B" to prevent position bias.

## EVALUATION CRITERIA
[EVALUATION_CRITERIA]

## OUTPUT A

[OUTPUT_A]

codeCopy

## OUTPUT B

[OUTPUT_B]

codeCopy

## INPUT CONTEXT (what both outputs were responding to)
[INPUT_CONTEXT]

## SOURCE MATERIAL (if applicable)
[SOURCE_MATERIAL]

## INSTRUCTIONS
1. Compare Output A and Output B against the evaluation criteria above.
2. Do not assume that longer outputs are better. Do not assume that more confident outputs are more correct.
3. If you detect any clues about which model or system produced either output (e.g., signature phrases like "As an AI language model," distinctive formatting, refusal patterns, or self-identification), flag them in the identity leakage section.
4. Provide your preference, justification, confidence score, and key differentiators in the specified JSON format.
5. If both outputs are equally good or equally flawed, you may declare a tie but must explain why neither output is clearly superior.

## OUTPUT FORMAT
Return ONLY valid JSON with this exact structure:
{
  "preference": "A" | "B" | "TIE",
  "confidence": 0.0-1.0,
  "justification": "Detailed explanation of why the preferred output is better, referencing specific criteria.",
  "key_differentiators": ["Differentiator 1", "Differentiator 2"],
  "identity_leakage_detected": true | false,
  "identity_leakage_details": "If leakage detected, describe what clues were found and in which output. If none, write 'None detected.'",
  "tie_reason": "If preference is TIE, explain why neither output is clearly superior. Otherwise null."
}

## CONSTRAINTS
[CONSTRAINTS]

To adapt this template for your pipeline, replace [EVALUATION_CRITERIA] with your specific rubric dimensions—such as accuracy, conciseness, safety, or instruction adherence—each with clear definitions and weights. The [OUTPUT_A] and [OUTPUT_B] placeholders should be populated by your harness after randomizing which output gets which label; never hardcode a consistent mapping. The [INPUT_CONTEXT] field should contain the original user query or task description that both outputs were responding to. If your evaluation requires grounding against source documents, populate [SOURCE_MATERIAL] with the relevant passages. The [CONSTRAINTS] placeholder lets you add domain-specific rules, such as word limits, required sections, or prohibited content categories. After populating all placeholders, validate that the prompt contains no residual model names, version identifiers, or system fingerprints that could unblind the evaluator. Run this prompt through your evaluation harness with position-swapped duplicates to measure position bias, and compare results against human preference labels to calibrate the judge's confidence scores.

The Blind A/B Output Evaluation Prompt is designed to be a single step inside a larger evaluation harness, not a standalone chat interaction. The harness is responsible for stripping model identity signals, randomizing output order, collecting the judge's preference and rationale, and then checking whether the evaluator was truly blind. Without this surrounding machinery, the prompt alone cannot guarantee unbiased comparisons—model names, stylistic fingerprints, or output formatting conventions can leak identity and bias the judge.

Wire the prompt into your application by building a pre-processing layer that accepts two raw outputs—Output A and Output B—along with the original user input and any shared context. Before calling the LLM judge, the harness must: (1) strip all model name mentions, version strings, and provider-specific formatting from both outputs using regex and string replacement rules; (2) randomly assign the two outputs to Position 1 and Position 2 using a seeded randomizer that logs the mapping for later decoding; (3) inject the position-randomized outputs into the [OUTPUT_A] and [OUTPUT_B] placeholders; and (4) append the [LEAKAGE_CHECK_INSTRUCTIONS] block that asks the judge to flag any detected model identity signals. After receiving the judge's response, run a post-processing validator that parses the structured preference verdict, extracts the confidence score, and checks the leakage flag. If leakage is detected, flag the comparison for human review or discard it from aggregate metrics. Log every comparison with the position mapping, judge identity, timestamp, and raw response for auditability.

For high-stakes model selection or RLHF data pipelines, add a second judge pass with reversed positions to measure position bias. If the same judge prefers Output X in Position 1 but Output Y when positions are swapped, the comparison is unreliable and should be excluded or escalated. Implement retry logic with exponential backoff for malformed JSON responses, but cap retries at three attempts before logging a failure and moving on. Use a structured output API or constrained decoding if your model provider supports it—this reduces parsing failures and makes the preference verdict machine-readable without regex extraction. Store all evaluation results in a schema that includes: comparison_id, judge_model, output_a_id, output_b_id, position_mapping, preference_verdict, confidence_score, rationale_summary, leakage_detected, leakage_details, position_bias_check_passed, and timestamp. This schema enables downstream analysis of judge reliability, position bias trends, and identity leakage rates across your evaluation pipeline.

When integrating this prompt into a CI/CD pipeline for model releases, set a minimum confidence threshold (e.g., 0.7) below which comparisons are treated as ties or escalated to human review. For RLHF data generation, only accept preference pairs where confidence exceeds 0.8 and no identity leakage was detected. Avoid using this prompt for single-output absolute quality scoring—it is designed for relative comparison only. If you need absolute pass/fail judgments, pair this with a rubric-based evaluation prompt from the Pass/Fail Criteria pillar. Finally, never expose raw model identities in the prompt context, even in comments or metadata fields—the leakage check can only catch what the harness fails to strip, and a single leaked model name can bias an entire evaluation batch.