Relevance Scoring Prompt Template

This prompt is designed for RAG and search teams who need to programmatically evaluate the relevance of an AI-generated answer against a user's query and the provided context. It acts as an LLM judge, producing a structured relevance score, a detailed justification, and flags for off-topic drift or context misuse. Use this when you are building an automated evaluation harness, running regression tests on retrieval pipelines, or need a repeatable metric for answer-to-query alignment before shipping a new prompt or model. It is not a replacement for human evaluation of tone or style, nor is it designed for open-ended creative writing tasks.

The ideal user is an AI quality engineer or platform engineer embedding this prompt into a CI/CD gate. You must supply three inputs: the user's original query, the retrieved context chunks, and the final generated answer. The prompt expects these to be clearly delimited so the judge does not confuse evidence with output. In practice, wire this into your existing eval harness by feeding it the same query-context-answer triples you already log. Run it against a golden dataset of known-relevant and known-irrelevant examples to calibrate your pass/fail thresholds before relying on it as a gate. Avoid using this prompt when the answer quality depends on subjective style, brand voice, or creative flair—those dimensions require separate human-aligned rubrics.

Do not deploy this prompt as a real-time user-facing feature. It is an internal evaluation tool. If your application operates in a regulated domain—such as healthcare, legal, or finance—the output of this judge is a signal, not a compliance check. Always pair automated relevance scores with human review for high-stakes decisions, and log every judgment with its query, context, and answer for auditability. Next, copy the prompt template into your eval harness and run it against at least 50 labeled examples to establish your baseline score distribution before setting any automated gates.

This prompt template is the core instruction set for an LLM judge tasked with evaluating retrieval-augmented generation (RAG) outputs. It is designed to be used as a system message in your evaluation harness, producing a structured relevance score that measures how well an answer aligns with the query, utilizes the provided context, and avoids off-topic drift. The template uses square-bracket placeholders for all dynamic inputs, making it ready to be wired into a scripted evaluation loop where you iterate over a golden dataset of query-context-answer triples.

textCopy
You are an expert relevance evaluator for a RAG system. Your job is to score the alignment between a user's query, the retrieved context, and the generated answer.

## INPUT
[QUERY]
[CONTEXT]
[ANSWER]

## SCORING CRITERIA
Evaluate the answer on a scale of 1 to 5 based on the following:
- **Score 5 (Perfect):** The answer directly addresses the query using only information from the context. It is concise and contains no extraneous information.
- **Score 4 (Good):** The answer addresses the query using context, but includes minor, harmless extra details or is slightly verbose.
- **Score 3 (Partial):** The answer partially addresses the query, but misses a key point, or relies on a mix of context and general knowledge.
- **Score 2 (Weak):** The answer is tangentially related to the query but fails to use the context effectively, or mostly ignores the context.
- **Score 1 (Irrelevant):** The answer does not address the query, directly contradicts the context, or is completely off-topic.

## DISTRACTION RESISTANCE
If the context contains information irrelevant to the query, a perfect answer must ignore it. Penalize answers that incorporate distracting context.

## OUTPUT FORMAT
Return ONLY a valid JSON object. Do not include any text outside the JSON block.
{
  "score": <integer 1-5>,
  "rationale": "<One-sentence explanation referencing the context and query.>"
}

## CONSTRAINTS
- Do not judge the truthfulness of the context itself, only the answer's use of it.
- If the answer is an empty string, the score must be 1.
- If the query is unanswerable from the context, a perfect score requires the answer to state that clearly.

To adapt this template, replace the [QUERY], [CONTEXT], and [ANSWER] placeholders with your test data at runtime. For high-stakes production gates, you should calibrate this judge against human-annotated scores using the LLM Judge Calibration Prompt Template to validate that its scoring distribution matches your quality bar. Always run this evaluation on a held-out golden dataset before promoting a new prompt version, and log the rationale field to debug scoring disagreements.

The relevance scoring prompt is designed to operate as a synchronous evaluation step within a larger RAG QA pipeline or a regression test suite. It should be called after retrieval and answer generation are complete, receiving the user query, the generated answer, and the retrieved context chunks as inputs. The prompt is stateless and idempotent for the same inputs, making it safe to retry on transient model errors. Wire it as a post-generation gate: if the relevance score falls below a configured threshold, the system can either flag the output for human review, discard it and regenerate, or log it for offline analysis.

Validation and output parsing are critical because the prompt returns a structured JSON block. Implement a post-processing validator that checks: (1) the response contains valid JSON, (2) the score field is a float between 0.0 and 1.0, (3) the partial_relevance array contains valid sub-scores if present, and (4) the distraction_resistance object includes a distracted boolean and a rationale string. If validation fails, retry up to two times with a slightly lowered temperature (e.g., 0.1 → 0.0) to reduce output variance. Log all validation failures and retry attempts with the raw model output for debugging. For high-stakes use cases, route scores in the ambiguous range (0.4–0.6) to a human reviewer queue rather than relying on automated pass/fail gating.

Model choice matters for consistency. Use a model with strong instruction-following and JSON output capabilities—GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro are suitable defaults. Avoid smaller or older models that may drift on the structured output format or produce inconsistent relevance judgments. If running at scale, consider caching scores for identical (query, answer, context) tuples to avoid redundant LLM calls. For regression test suites, store all scores alongside the prompt version, model version, and timestamp to enable trace-based analysis and drift detection over time. Never use this prompt as the sole quality gate for production RAG systems—pair it with factual accuracy checks, citation verification, and user-facing feedback signals.