LLM Judge Prompt for Passage Relevance Comparison

This prompt is built for evaluation engineers and AI quality teams who need to run head-to-head comparisons between two passage filtering approaches, model versions, or retrieval configurations. Instead of relying on aggregate metrics like recall@k or NDCG alone, this prompt produces a structured, comparative relevance judgment with detailed reasoning for each query-passage pair. The core job-to-be-done is generating an auditable, human-readable trail that explains why one filtering strategy outperforms another on specific examples. Use it when you are running A/B filter evaluations, calibrating a new relevance scorer against a baseline, or generating evidence for inter-rater reliability checks and bias detection workflows. The prompt forces the model to justify its preference, flag ties, and surface potential biases in its own judgment, which makes it suitable for detecting systematic failures that aggregate metrics often hide.

This prompt assumes you already have two ranked or filtered passage sets for the same query and need a comparative judgment, not an absolute relevance score. The required inputs are the original query, the two passage sets (labeled Set A and Set B), and any optional evaluation criteria or rubrics you want the judge to apply. The expected output is a structured JSON object containing a preference decision (A, B, or tie), a confidence score, a detailed justification, and a bias self-check. Do not use this prompt for single-system relevance scoring, binary filtering, or tiered labeling—those tasks have dedicated playbooks in this content group. Using this prompt for absolute relevance assessment will produce misleading results because the model is instructed to compare, not to score in isolation. The prompt is also not suitable for production latency-sensitive filtering pipelines; it is an offline evaluation tool designed for depth, not speed.

Before running this prompt at scale, calibrate it against a golden dataset of human-annotated query-passage pairs where the correct preference is known. Measure the judge's agreement rate with human evaluators and check for position bias (favoring Set A or Set B based on presentation order) by randomizing the set labels. If the judge shows systematic bias toward longer passages, more fluent text, or certain source domains, document those biases in your evaluation report rather than treating the LLM judgments as ground truth. For high-stakes filtering decisions—such as medical, legal, or financial content—always include human review of a statistically significant sample of the judge's outputs. The next step after reading this section is to review the prompt template and implementation harness to understand how to wire this judge into your existing evaluation pipeline.

This prompt template is designed for evaluation engineers who need to compare the output of two different passage filtering systems, model versions, or prompt variants against the same query and retrieval set. It produces a structured comparison that includes relevance judgments, reasoning for each decision, and explicit flags for agreement, disagreement, and potential bias. Use this when you need reproducible, side-by-side evaluations to decide which filtering approach performs better before deploying to production. Do not use this for single-system evaluation or when you lack a baseline to compare against—use a single-system scoring prompt instead.

textCopy
You are an impartial evaluation judge comparing two passage filtering systems (System A and System B) on the same retrieval task. Your role is to produce structured, evidence-based comparative judgments that can be used for A/B evaluation and inter-rater reliability analysis.

## INPUT

**Query:** [QUERY]

**Retrieved Passages (pre-filtering):**
[RETRIEVED_PASSAGES]

**System A Filtered Passages:**
[SYSTEM_A_OUTPUT]

**System B Filtered Passages:**
[SYSTEM_B_OUTPUT]

**Evaluation Criteria:**
[EVALUATION_CRITERIA]

## TASK

For each passage in the original retrieval set, produce a comparative judgment following the output schema below. You must:

1. Assess whether each passage is truly relevant to the query (ground truth judgment).
2. Compare System A and System B decisions against that ground truth.
3. Identify agreements, disagreements, and potential bias patterns.
4. Provide detailed reasoning for each comparative judgment.

## OUTPUT SCHEMA

Return a JSON object with the following structure:

```json
{
  "query": "string",
  "evaluation_date": "ISO 8601 timestamp",
  "passage_evaluations": [
    {
      "passage_id": "string",
      "passage_text": "string",
      "ground_truth_relevance": "relevant | irrelevant | partially_relevant",
      "ground_truth_reasoning": "string",
      "system_a_decision": "kept | filtered",
      "system_b_decision": "kept | filtered",
      "comparative_judgment": "agreement_correct | agreement_incorrect | a_better | b_better | both_wrong",
      "comparative_reasoning": "string",
      "confidence": 0.0-1.0
    }
  ],
  "aggregate_metrics": {
    "system_a_precision": 0.0-1.0,
    "system_a_recall": 0.0-1.0,
    "system_b_precision": 0.0-1.0,
    "system_b_recall": 0.0-1.0,
    "agreement_rate": 0.0-1.0,
    "winner": "a | b | tie"
  },
  "bias_detection": {
    "position_bias_detected": true/false,
    "position_bias_evidence": "string or null",
    "length_bias_detected": true/false,
    "length_bias_evidence": "string or null",
    "other_biases": ["string"]
  },
  "inter_rater_notes": "string explaining any edge cases or judgment calls that another evaluator might decide differently"
}

CONSTRAINTS

• Judge each passage independently before computing aggregate metrics.
• If a passage is partially relevant, explain what portion is relevant and what is not.
• Flag any position bias (e.g., System A always listed first) or length bias in your reasoning.
• Use confidence scores to indicate judgment certainty, not model confidence.
• If the query is ambiguous, state your interpretation in the inter_rater_notes field.
• Do not favor either system by default. Base all judgments on evidence.

To adapt this prompt for your evaluation pipeline, replace each square-bracket placeholder with actual data before sending to the model. The [QUERY] should be the original user question or search intent. [RETRIEVED_PASSAGES] should contain the full set of passages before any filtering, with unique identifiers for each passage. [SYSTEM_A_OUTPUT] and [SYSTEM_B_OUTPUT] should contain the filtered passage sets from the two systems you are comparing, using the same passage identifiers. [EVALUATION_CRITERIA] should specify what makes a passage relevant for this specific use case—be explicit about relevance definitions, as vague criteria produce unreliable judgments. If your provider supports structured output mode, use it to enforce the JSON schema directly. Otherwise, parse the JSON block from the response and validate it against the expected schema before using the results. For high-stakes evaluations where filtering errors could cause downstream harm, always run this prompt with multiple LLM judges and compute inter-rater agreement scores before trusting the comparison.

This prompt is designed to operate inside an automated evaluation harness, not as a one-off manual comparison. The primary integration point is a comparison function that receives two ranked passage lists (from different retrieval or filtering strategies) and a shared query. The harness should call the LLM Judge with a structured prompt, parse the JSON output, and log the comparative judgment along with the detailed reasoning. This structured output is then consumed by downstream analytics to compute win rates, statistical significance, and bias metrics across your evaluation dataset. The harness must enforce a strict schema on the model's response to prevent parsing failures in automated pipelines.

To implement this, wrap the prompt template in a function that accepts query, passages_a, and passages_b as arguments. Before calling the model, validate that both passage lists are non-empty and contain distinct items. Set the model's response_format to JSON mode (or use a structured output API) and provide the expected schema: an object with winner (enum: 'A', 'B', 'tie'), confidence (0.0-1.0), reasoning (string), and bias_flags (array of strings). Implement a retry loop with up to 3 attempts for malformed JSON or schema violations, logging each failure. For high-stakes comparisons, route tie results and low-confidence judgments (confidence < 0.7) to a human review queue. Store the raw prompt, full response, and parsed judgment in an evaluation database with timestamps for later analysis.

Inter-rater reliability checks require running the same comparison multiple times (typically 3-5 runs) with the temperature set to a non-zero value (e.g., 0.2) to surface judgment instability. The harness should compute Cohen's kappa or Krippendorff's alpha across runs and flag query-passage pairs where agreement falls below 0.6 for manual inspection. For bias detection, track win rates per system across demographic or topic slices of your evaluation set. If one system consistently wins for certain query types, the harness should alert the team to potential evaluation bias rather than true quality differences. Avoid running this judge on single-passage comparisons; the prompt is calibrated for ranked lists and may produce unreliable results with insufficient context.