Pairwise Evidence Comparison Scoring Prompt

This prompt is designed for evaluation engineers and RAG pipeline architects who need to build fine-grained re-ranking systems or calibration datasets. The core job-to-be-done is comparing two evidence passages head-to-head against a specific query to produce a relative quality judgment, not an absolute score. It is ideal when you need to resolve close ranking decisions, detect subtle quality differences that absolute scoring models miss, or generate preference data for training or evaluating your own scoring models.

Use this prompt when you have already retrieved a candidate set and need to establish a precise partial ordering. It is particularly effective for creating golden datasets for LLM-as-a-Judge evaluators, calibrating production re-rankers, and auditing retrieval quality where the difference between the 3rd and 4th result matters. The prompt requires a clear query, two evidence passages formatted with source identifiers, and a defined set of comparison criteria such as relevance, specificity, authority, and recency. You must also provide an output schema that captures the winner, the margin of victory, and a structured justification.

Do not use this prompt for bulk scoring of dozens of passages against a query; a pointwise scoring prompt is more efficient for that task. Avoid it when passages are fundamentally incommensurable—comparing a peer-reviewed paper to a tweet about the same topic will produce a trivial judgment that wastes inference compute. This prompt is also insufficient on its own for production ranking systems: you must implement transitivity checks across multiple pairwise comparisons, position bias detection by swapping passage order, and human calibration of the resulting preference pairs. The output of this prompt is a building block for a larger evaluation or training pipeline, not a finished product feature.

This template implements pairwise evidence comparison scoring, where two evidence passages are evaluated side by side against a query to determine which provides stronger support. The prompt is designed for evaluation engineers building precision re-ranking systems and calibration datasets. It produces relative quality judgments with explicit reasoning, making it suitable for fine-grained evidence ranking, LLM-as-judge pipelines, and training data generation for learned re-ranking models. The template includes structural safeguards against common failure modes: position bias (favoring the first or second passage regardless of quality), length bias (favoring longer passages), and superficial relevance (favoring keyword overlap over semantic alignment).

codeCopy
You are an evidence quality evaluator. Your task is to compare two evidence passages and determine which one better supports answering a given query.

## QUERY
[QUERY]

## PASSAGE A
[PASSAGE_A]

## PASSAGE B
[PASSAGE_B]

## COMPARISON CRITERIA
Evaluate both passages on these dimensions, in order of importance:

1. **Relevance**: How directly does the passage address the query's information need? Consider semantic alignment, not just keyword overlap.
2. **Specificity**: Does the passage provide concrete details, facts, or data that directly bear on the query? Penalize vague or general statements.
3. **Authority**: Based on available signals (source type, date, author indicators), how credible is this passage for this query domain?
4. **Completeness**: How much of the query's information need does this passage address on its own?

## OUTPUT FORMAT
Return a JSON object with this exact schema:

```json
{
  "winner": "A" | "B" | "TIE",
  "confidence": 0.0-1.0,
  "reasoning": {
    "passage_a_strengths": ["specific strength 1", "specific strength 2"],
    "passage_a_weaknesses": ["specific weakness 1"],
    "passage_b_strengths": ["specific strength 1"],
    "passage_b_weaknesses": ["specific weakness 1", "specific weakness 2"],
    "decisive_factor": "explanation of what most determined the outcome"
  },
  "position_bias_check": {
    "swapped_expected_winner": "A" | "B" | "TIE",
    "explanation": "If the passages were swapped, would the winner change? Explain why or why not."
  }
}

CONSTRAINTS

• Do not favor a passage because it appears first or second. The position_bias_check field forces you to verify this.
• Do not favor longer passages. A short, precise passage can outrank a long, diffuse one.
• If both passages are equally strong or equally weak, return "TIE" with confidence reflecting how close the comparison was.
• If neither passage is relevant to the query, return "TIE" with low confidence and explain why both are inadequate.
• Base your judgment only on the passage content provided. Do not bring in external knowledge.
• For the confidence score: use 0.9+ for clear winners, 0.7-0.9 for moderate advantage, 0.5-0.7 for close calls, below 0.5 for uncertain judgments.

EXAMPLES

[EXAMPLES]

To adapt this template for your pipeline, start by populating the [QUERY], [PASSAGE_A], and [PASSAGE_B] placeholders with your retrieval results. The [EXAMPLES] placeholder should contain 2-4 annotated pairwise comparisons that demonstrate your desired judgment standards, including at least one TIE case and one case where the shorter passage wins. For production use, add a [RISK_LEVEL] parameter that gates whether low-confidence results (below 0.6) require human review. If your domain has specific authority signals (e.g., peer-reviewed sources for scientific queries, recency for news queries), add those as explicit sub-criteria under the Authority dimension. The position_bias_check field serves dual purpose: it forces the model to self-audit for position bias and provides a signal you can use in post-processing to detect systematic bias patterns across many comparisons. For transitivity validation, run the same passage pair in both A/B and B/A orderings and verify that the winner is consistent after accounting for the position swap.

The Pairwise Evidence Comparison Scoring Prompt is designed to be called repeatedly in a tournament-style ranking loop, not as a single one-shot request. In practice, you will iterate over all unique pairs from a retrieved evidence set, calling the prompt for each pair to produce a relative quality judgment. The harness must manage state across these calls, aggregate the head-to-head results into a global ranking, and apply transitivity checks to detect inconsistent judgments before the scores are trusted downstream. This prompt is a building block for fine-grained re-ranking and calibration dataset creation, so the harness is responsible for ensuring the comparisons are fair, unbiased, and auditable.

Wire the prompt into your application as a stateless function that accepts two evidence passages and a query, then returns a structured JSON object with the winner, confidence, and reasoning. The harness should randomize the order of passages in each call to mitigate position bias—swap passage A and B positions in 50% of calls and normalize the output so the winner field always references the original passage IDs. Implement a validation layer that checks the returned JSON schema, confirms the winner is one of the two provided passage IDs, and rejects any response where the confidence score falls below a configurable threshold (start at 0.6 for high-stakes ranking). Log every comparison with a unique pair ID, the raw model response, and the validated output for audit trails and debugging. For model choice, use a model with strong reasoning capabilities (such as GPT-4o or Claude 3.5 Sonnet) and set temperature to 0 to maximize consistency across repeated comparisons. If you are processing large retrieval sets, batch the comparisons and implement a retry queue with exponential backoff for any pair that fails validation—do not silently drop failed comparisons or you will introduce ranking gaps.

After all pairwise comparisons complete, aggregate the results into a win-loss matrix and compute a global ranking using a simple win-count or Elo-style scoring system. Run transitivity checks: if A beats B and B beats C, verify that A beats C in the direct comparison. Flag any transitivity violations for human review or automatic re-scoring with a higher-confidence threshold. For calibration dataset creation, store the aggregated rankings alongside human-judgment baselines to measure alignment over time. Avoid wiring this prompt directly into user-facing latency-sensitive paths—pairwise comparison is inherently O(n²) and should run asynchronously in evaluation or offline re-ranking pipelines. The most common production failure is position bias leaking through despite randomization; monitor win rates by position slot to detect when the model favors the first passage regardless of content.