RAG Passage Relevance Scoring Prompt Template

This prompt is for RAG pipeline engineers who need to score the relevance of retrieved passages against a user query before those passages are injected into the generation context. The core job-to-be-done is calibrated filtering: converting a noisy retrieval set into a ranked, scored list where each passage receives a relevance score and a human-readable justification. The ideal user is an engineer or ML operator tuning a production retrieval-augmented generation system, not an end-user chatting with a bot. You should use this prompt when your retrieval step returns more passages than you can safely pack into a context window, when you need an auditable relevance signal to trigger fallback or human review, or when you are calibrating a downstream reranker and need a baseline LLM relevance judgment to compare against.

This prompt is not a replacement for a trained cross-encoder reranker when latency budgets are under 100ms or when you are scoring thousands of passage pairs per query. It is also not the right tool when you need a hard binary include/exclude decision—use the Binary Passage Relevance Filter Prompt Template for that. Do not use this prompt if your passages contain highly sensitive or regulated data that cannot be sent to an external model endpoint without redaction. The prompt assumes you have already retrieved passages and that each passage has a stable identifier you can reference in the output. You must provide the original user query, the set of candidate passages with their IDs, and a defined relevance scale with clear criteria for each score level.

Before integrating this prompt into a production pipeline, you must calibrate the scoring rubric against a golden dataset of query-passage pairs with human relevance judgments. Run the prompt over your eval set, measure score distribution alignment, and tune the scale descriptions until inter-rater agreement between the LLM and human annotators reaches an acceptable threshold for your use case. If your application is in a regulated domain—such as healthcare, legal, or finance—add a human review step for passages that fall into a 'borderline' score bucket, and log every scored passage with its justification for audit. The next section provides the copy-ready prompt template you can adapt and deploy.

This prompt template is designed for RAG pipeline engineers who need to score retrieved passages before they are injected into the generation context. The template forces the model to produce a calibrated relevance score and a concise justification for each passage, making the selection step auditable and tunable. Use it as a filtering gate: passages scoring below a defined threshold should be discarded or flagged for human review before answer generation.

textCopy
You are a relevance scoring engine for a retrieval-augmented generation (RAG) system. Your task is to evaluate each retrieved passage against the user query and assign a relevance score along with a brief justification.

## INPUT
User Query: [USER_QUERY]
Retrieved Passages:
[PASSAGE_LIST]

## SCORING CRITERIA
Score each passage on a scale of [SCORE_MIN] to [SCORE_MAX] using these criteria:
- [SCORE_HIGH]: The passage directly answers or contains key information needed to answer the query.
- [SCORE_MEDIUM]: The passage is topically related but lacks specific answer details.
- [SCORE_LOW]: The passage is only tangentially related or contains no useful information.
- [SCORE_IRRELEVANT]: The passage is unrelated to the query.

## CONSTRAINTS
- Do not generate an answer to the user query. Only score relevance.
- If a passage is ambiguous, score conservatively and note the ambiguity in the justification.
- If the passage contains contradictory information relative to other passages, flag it with a `conflict_flag: true`.
- Ignore passage order; treat each independently.

## OUTPUT SCHEMA
Return a valid JSON object with this exact structure:
{
  "query": "string",
  "passages": [
    {
      "passage_id": "string",
      "relevance_score": number,
      "justification": "string",
      "conflict_flag": boolean
    }
  ]
}

## EXAMPLES
[FEW_SHOT_EXAMPLES]

## RISK LEVEL
[RISK_LEVEL]

Adaptation guidance: Replace [SCORE_MIN] and [SCORE_MAX] with your numeric range (e.g., 0-1 or 1-5). Define [SCORE_HIGH] through [SCORE_IRRELEVANT] with concrete descriptions that match your domain's tolerance for noise. The [PASSAGE_LIST] placeholder should be populated with pre-numbered or ID-tagged passages from your retrieval step. Include 2-3 [FEW_SHOT_EXAMPLES] that demonstrate borderline cases and conflict scenarios. Set [RISK_LEVEL] to high if downstream actions are irreversible; this should trigger additional validation and human review gates in your harness. Always validate the output JSON against the schema before passing scores to the next pipeline stage. If the model returns malformed JSON, use a repair prompt or retry with stricter formatting instructions.