Tiered Passage Relevance Labeling Prompt

This prompt is for RAG system builders who need to sort retrieved passages into high, medium, and low relevance tiers before context assembly. The job-to-be-done is reducing noise and hallucination by ensuring that only appropriately relevant passages influence the final answer. The ideal user is an engineering lead or AI pipeline developer who already has a retrieval step producing candidate passages and needs a structured, explainable filtering layer before generation. Required context includes the user query, the set of retrieved passages, and any domain-specific relevance criteria that define what 'high relevance' means for your application.

Do not use this prompt when you need a simple binary relevant/irrelevant decision—use the Binary Passage Relevance Filter Prompt instead. Do not use it when you need a fully ranked, ordered list of passages; a cross-encoder reranking prompt is more appropriate for that task. This prompt is also not a substitute for retrieval quality. If your retriever consistently returns irrelevant results, tiered labeling will only document the failure rather than fix it. The prompt works best when the retrieval set already contains a mix of highly relevant, partially relevant, and irrelevant passages, and you need to separate them with clear, auditable criteria before context window packing or answer generation.

Before deploying this prompt, define your tier definitions precisely. A passage that is 'high relevance' should directly answer the query with specific evidence. 'Medium relevance' passages should provide useful context or partial answers but lack the specificity to stand alone. 'Low relevance' passages should be tangentially related or contain only background information. Write these definitions into the [TIER_DEFINITIONS] placeholder and test them against a golden dataset of query-passage pairs with known relevance labels. If your application operates in a regulated domain such as healthcare, finance, or legal, add a human review step for any passage labeled 'high relevance' that will be used to generate user-facing claims. The next section provides the copy-ready template you can adapt.

The following prompt template is designed to be copied directly into your prompt library or orchestration code. It uses square-bracket placeholders for all dynamic inputs, allowing you to swap in your own query, passages, tier definitions, and output schema without rewriting the core instruction logic. The template enforces structured JSON output and requires the model to explain its tier assignment for each passage, which is critical for downstream debugging and auditability in production RAG pipelines.

textCopy
You are a relevance labeling assistant for a retrieval-augmented generation (RAG) system. Your task is to evaluate each retrieved passage against the user query and assign a relevance tier: HIGH, MEDIUM, or LOW.

## TIER DEFINITIONS
- HIGH: The passage directly answers or substantially addresses the query. It contains specific facts, entities, or reasoning that are necessary to answer correctly.
- MEDIUM: The passage is topically related but does not directly answer the query. It provides useful context, background, or adjacent information that could support a broader answer.
- LOW: The passage is irrelevant, off-topic, or too vague to contribute meaningfully to answering the query.

## INPUT
**User Query:** [USER_QUERY]

**Retrieved Passages:**
[PASSAGES]

## OUTPUT SCHEMA
Return a valid JSON object with the following structure:
{
  "passages": [
    {
      "passage_id": "string",
      "tier": "HIGH | MEDIUM | LOW",
      "criteria_explanation": "string explaining why this tier was assigned, referencing specific content in the passage and its relationship to the query"
    }
  ]
}

## CONSTRAINTS
- Assign exactly one tier per passage.
- Every passage in the input must appear in the output.
- The criteria_explanation must be specific and evidence-based, not generic.
- Do not reorder passages; preserve the original input order.
- If a passage is ambiguous, default to MEDIUM and explain the ambiguity.
- Do not answer the user query. Only label relevance.

To adapt this template for your own system, replace [USER_QUERY] with the end user's question or search string, and [PASSAGES] with your retrieved passage set formatted as a list of objects containing at minimum a passage_id and text field. If your retrieval pipeline includes metadata such as source titles, publication dates, or authority scores, append those fields to each passage object in the input—the model can use them to improve tiering accuracy. For high-stakes domains such as legal or medical RAG, add a [RISK_LEVEL] placeholder and a constraint requiring the model to flag passages where tier assignment is uncertain and human review is recommended. Always validate the output JSON against the schema before passing tiered passages to your answer generation step; malformed outputs should trigger a retry or fallback to a default tier assignment.

The tiered labeling prompt is designed to sit between retrieval and context assembly in a RAG pipeline. It accepts a batch of retrieved passages and a user query, then returns a structured relevance tier for each passage along with a criteria explanation. The output is not a final answer—it is a filtering and prioritization signal that downstream components consume. In a typical implementation, you will call this prompt once per query after initial retrieval, passing the top-k passages (often 20–50) and the original user query. The resulting tier labels feed into a context assembly step that selects passages by tier priority, respecting token budgets and diversity constraints.

Integration pattern. Wrap the prompt in a function that accepts query: str, passages: list[dict], and optional tier_thresholds: dict. The function should construct the prompt by injecting the query and passages into the template, call the model with response_format set to a JSON schema matching the expected output, and parse the result. Validate the output immediately: check that every passage ID in the input appears exactly once in the output, that each label is one of HIGH, MEDIUM, or LOW, and that every entry includes a non-empty criteria string. If validation fails, retry once with the validation errors appended to the prompt as feedback. If the retry also fails, log the failure and route the query to a fallback path—either using all passages unfiltered or escalating for human review if the use case is high-stakes.

Model choice and latency. For most production RAG systems, a fast instruction-tuned model like GPT-4o-mini, Claude Haiku, or Gemini Flash is sufficient for tiered labeling. The task is classification with explanation, not generation, so smaller models often perform well. If you are processing passages in parallel across multiple queries, consider batching requests or using asynchronous calls to keep end-to-end latency under 500ms. For high-throughput systems, implement a caching layer that stores tier labels keyed by (query_hash, passage_hash) to avoid re-labeling identical query-passage pairs. Monitor label distribution drift over time—if the ratio of HIGH to LOW passages shifts unexpectedly, it may indicate retrieval quality degradation or prompt drift.

Eval harness and human review gates. Before deploying, build an eval harness that compares model-assigned tiers against a golden dataset of 200–500 query-passage pairs labeled by domain experts. Measure tier agreement using Cohen's kappa or a weighted F1 that penalizes HIGH/LOW confusion more than HIGH/MEDIUM. Track boundary cases where the model disagrees with human labels by more than one tier—these are your highest-risk failures. In regulated or high-stakes domains (healthcare, legal, finance), route all LOW-tier passages that the model recommends for exclusion to a human review queue before they are dropped from context. Log every tiering decision with the query, passage ID, assigned tier, criteria, and model version for auditability. Avoid using tier labels as the sole gate for critical information without a human-in-the-loop fallback.