LLM-as-Judge Prompt for Context Relevance

This prompt is for eval pipeline builders who need to measure how well their retrieval system returns context that actually helps answer a question. It uses an LLM judge to score each retrieved passage against a query and a gold-standard answer, producing calibrated relevance scores. Use this prompt when you are tuning retrieval parameters, comparing embedding models, setting noise-filter thresholds, or building regression tests that catch retrieval degradation. This is not a prompt for end users. It is infrastructure for your evaluation harness.

The prompt assumes you already have a set of queries, gold answers, and retrieved passages. It does not perform retrieval itself, and it does not generate answers. It judges the relevance of what retrieval already returned. You should wire this into a batch evaluation script that iterates over your test dataset, calls the LLM judge for each query-passage pair, and collects scores for analysis. The output is a structured relevance score per passage, not a final answer. This separation keeps your evaluation logic clean: retrieval quality is measured independently from generation quality.

Do not use this prompt when you lack gold-standard answers, when you need real-time retrieval decisions in a production path, or when you are evaluating end-to-end answer quality rather than retrieval quality. This prompt is also inappropriate for datasets where relevance is inherently subjective without clear ground truth—the judge needs a gold answer to calibrate against. If your evaluation requires pairwise comparison between retrieval systems rather than absolute relevance scoring, use a pairwise judge prompt instead. For production guardrails that filter context before generation, use the Context Quality Gate Evaluation Prompt or Retrieval Noise Classification Prompt Template, not this eval-focused judge.

This prompt template turns an LLM into a calibrated relevance judge for your RAG retrieval pipeline. It evaluates each retrieved passage against a user query and an optional gold-standard answer, producing a structured relevance score, a concise rationale, and a flag for passages that are misleading distractors. The output is designed to be machine-readable so you can aggregate scores across your eval dataset, tune retrieval thresholds, and catch regressions before they reach users.

textCopy
You are a strict retrieval relevance judge. Your task is to evaluate how relevant each retrieved passage is to answering the given query.

## INPUT
Query: [QUERY]
Gold Answer (for calibration, optional): [GOLD_ANSWER]
Retrieved Passages:
[PASSAGES]

## EVALUATION CRITERIA
For each passage, assign a relevance score from 1 to 5 using this rubric:
- 5 (Directly Relevant): The passage contains information that directly answers the query or provides a key piece of evidence needed to construct the answer.
- 4 (Supporting Context): The passage provides useful background, definitions, or supporting details that help contextualize the answer but does not directly answer the query on its own.
- 3 (Tangentially Related): The passage shares a topic or keywords with the query but does not contribute meaningful information toward answering it.
- 2 (Noise): The passage is from the right general domain but is irrelevant to the specific query. It would distract or dilute answer quality if included.
- 1 (Misleading Distractor): The passage appears relevant but contains information that contradicts the gold answer, is factually incorrect for this query, or would actively mislead answer generation.

## CONSTRAINTS
- Score every passage independently. Do not compare passages to each other.
- If a gold answer is provided, use it to calibrate your judgment, especially for identifying distractors (score 1).
- If no gold answer is provided, rely solely on query-passage alignment.
- Provide a one-sentence rationale for each score that cites the specific evidence or lack thereof.
- Flag any passage that is a near-duplicate of another with higher relevance. Note the duplicate's ID.

## OUTPUT_SCHEMA
Return a JSON object with this exact structure:
{
  "passage_scores": [
    {
      "passage_id": "string",
      "relevance_score": integer (1-5),
      "rationale": "string (one sentence)",
      "is_distractor": boolean,
      "duplicate_of": "string | null"
    }
  ],
  "aggregate": {
    "average_relevance": float,
    "distractor_count": integer,
    "noise_ratio": float
  }
}

## EXAMPLES
[EXAMPLES]

## RISK_LEVEL
[RISK_LEVEL]

Adapt this template by replacing the square-bracket placeholders with your data. [QUERY] takes the user's original question. [GOLD_ANSWER] is optional—include it when you have verified answers to calibrate distractor detection. [PASSAGES] should be a list of objects, each with an id and text field. [EXAMPLES] can contain 2-4 scored passage examples that demonstrate your specific relevance standards and edge cases. [RISK_LEVEL] accepts low, medium, or high—for high-risk domains like healthcare or legal, add an instruction requiring the judge to flag any passage where relevance is uncertain and recommend human review. After pasting this into your eval harness, validate the JSON output against the schema before recording scores. Run this judge across your golden eval set, track score distributions over time, and set automated alerts when the noise ratio or distractor count crosses your quality threshold.

This prompt is designed to operate as a deterministic scoring step inside an automated evaluation pipeline, not as a one-off manual check. The typical integration pattern places it after retrieval and before answer generation in a RAG eval harness, or as a standalone context-quality gate in a regression suite. The judge prompt expects three inputs: a query, a gold-standard answer, and a list of retrieved passages. It returns a structured JSON object containing a relevance score and justification for each passage, making it directly consumable by downstream metric aggregators, threshold monitors, and CI checks.

To integrate this into a production eval pipeline, wrap the prompt in a function that accepts the query, gold answer, and passage list as arguments. Validate the output against a strict JSON schema before accepting it—reject any response where scores fall outside the 1-5 range, justifications are missing, or the passage count doesn't match the input. Implement a retry loop with a maximum of two attempts using a lower-temperature fallback (e.g., temperature=0.1 on retry). Log every judge call with the input query hash, passage IDs, raw scores, and any validation failures. For regression testing, store a golden dataset of query-answer-passage triples with expected score ranges, and run this judge on every prompt or retrieval change. Flag any passage whose score deviates from the expected range by more than one point for human review.

Model choice matters for judge consistency. Use a model with strong instruction-following and JSON output capabilities—GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro are suitable defaults. Avoid smaller or older models that drift toward midpoint scores or produce malformed JSON under the scoring rubric. Set temperature to 0 for production eval runs to maximize reproducibility. If you're scoring thousands of passages per run, batch the calls but keep each batch independent—never let the judge see scores from other passages in the same request, as this introduces anchoring bias. For high-stakes pipelines where a bad relevance score could suppress critical evidence, add a human review step for passages scored 1 or 2 that the gold answer explicitly references.