Evidence Relevance Tier Assignment Prompt

This prompt is designed for search architects and RAG pipeline engineers who need to move beyond flat, numerical relevance scores and instead assign categorical relevance tiers to retrieved evidence passages. The core job-to-be-done is explicit, auditable filtering: before answer generation, you need to bucket passages into discrete tiers such as directly-supporting, contextually-relevant, tangentially-related, and irrelevant. This categorical approach makes downstream decisions—like context window allocation, citation selection, and answer abstention—deterministic and explainable. Use this prompt when your retrieval system returns a batch of passages and you require consistent, tiered assignments that can be reviewed, logged, and used to control how much context each passage contributes to the final prompt.

The ideal user is an engineer or architect who already has a working retrieval pipeline but struggles with inconsistent or opaque passage selection. You should use this prompt when you have a defined set of tier boundaries that align with your product's tolerance for noise and hallucination. For example, a medical RAG system might route only directly-supporting passages to the answer generator, while a research assistant might include contextually-relevant passages with a caveat. This prompt is not a replacement for initial retrieval or embedding-based semantic search. It is a post-retrieval, pre-generation classification step. You must provide the prompt with a batch of passages, the user query, and your organization's tier definitions. If you cannot articulate clear, mutually exclusive tier boundaries, this prompt will produce inconsistent results.

Do not use this prompt when you need a single ranked list of passages by a continuous score. For that, use a numerical relevance scoring prompt. Do not use it when your retrieval set is so large that per-passage classification becomes cost-prohibitive; in that case, apply a cheaper filtering step first. Do not use it when the downstream system cannot consume categorical labels or when you lack a mechanism to enforce tier-based filtering in code. Finally, avoid this prompt if your tier definitions are vague or overlap significantly—ambiguity in tier boundaries is the primary failure mode. Before implementing, define your tiers with concrete, testable criteria, and plan to run consistency checks across retrieval batches to ensure the model is applying your definitions reliably.

This template assigns each retrieved passage to one of four relevance tiers: directly-supporting, contextually-relevant, tangentially-related, or irrelevant. It is designed to be dropped into a retrieval re-ranking or filtering step before answer generation. The prompt enforces structured JSON output with explicit tier definitions and requires the model to justify each assignment, making the bucketing auditable and debuggable. Use this when you need categorical evidence filtering rather than continuous scores, and when downstream components need to know not just what is relevant but how relevant each passage is.

textCopy
You are an evidence relevance classifier for a retrieval-augmented generation system. Your task is to assign each provided passage to exactly one relevance tier based on how directly it supports or relates to the given query.

## TIER DEFINITIONS
- **directly-supporting**: The passage contains specific facts, data, or claims that directly answer or substantiate the query. It provides concrete information that would appear in a well-grounded answer.
- **contextually-relevant**: The passage provides background, definitions, related concepts, or broader context that helps understand the query domain but does not directly answer it. It would be useful as supplementary context.
- **tangentially-related**: The passage mentions the query topic or adjacent concepts but does not provide useful information for answering the query. It shares keywords but lacks substantive connection.
- **irrelevant**: The passage has no meaningful connection to the query. It may share incidental words but addresses a different topic entirely.

## INPUT
**Query:** [QUERY]

**Passages to classify:**
[PASSAGES]

## OUTPUT SCHEMA
Return a JSON object with the following structure:
{
  "classifications": [
    {
      "passage_id": "string",
      "tier": "directly-supporting | contextually-relevant | tangentially-related | irrelevant",
      "justification": "One-sentence explanation referencing specific passage content that supports this tier assignment.",
      "key_evidence_snippet": "Short quote or paraphrase from the passage that most influenced this classification, or null if irrelevant."
    }
  ],
  "tier_summary": {
    "directly_supporting_count": integer,
    "contextually_relevant_count": integer,
    "tangentially_related_count": integer,
    "irrelevant_count": integer
  }
}

## CONSTRAINTS
- Every passage in the input must appear exactly once in the output.
- Assign each passage to the highest tier that accurately describes its relationship to the query. Do not inflate tiers.
- If a passage could arguably fit two tiers, choose the higher one and note the ambiguity in the justification.
- The justification must reference specific content from the passage, not generic statements.
- If no passages are directly-supporting, set directly_supporting_count to 0 and do not fabricate support.
- Preserve the original passage_id exactly as provided in the input.

## EXAMPLES
[EXAMPLES]

## RISK LEVEL
[RISK_LEVEL]

Adaptation guidance: Replace [QUERY] with the user's original question or search intent statement. [PASSAGES] should be a JSON array of objects, each containing at minimum a passage_id and text field. The [EXAMPLES] placeholder should contain 2-4 few-shot examples showing correct tier assignments with edge cases such as near-miss passages that share keywords but are irrelevant, and passages that provide context without direct support. Set [RISK_LEVEL] to high for regulated domains where misclassification could cause harm, which will trigger additional conservatism in tier assignment and require human review of directly-supporting classifications. For production use, validate the output JSON against the schema before passing classifications downstream, and log any passages where the model's justification contradicts the assigned tier for pipeline debugging.

The tier assignment prompt is designed to sit between retrieval and answer generation in a RAG pipeline. It consumes a query and a batch of retrieved passages, then outputs a structured tier label for each passage. The implementation harness must handle batching, output validation, tier boundary consistency, and fallback behavior when the model produces ambiguous or malformed tier assignments. This prompt is not a standalone classifier; it is a pipeline component that requires surrounding application logic to be production-safe.

Batching and Throughput: Retrieved passages should be grouped into batches of 5–15 per prompt call to balance latency and context window limits. Each batch must include the original query, a unique passage identifier, and the passage text. The application layer should track passage IDs across batches to reconstruct the full tiered retrieval set. For high-throughput systems, parallelize batch calls across multiple model instances and merge results using the passage ID as the join key. Output Validation: Parse the model response as JSON and validate that every passage ID in the input batch appears exactly once in the output, that each tier label matches the allowed enum (directly-supporting, contextually-relevant, tangentially-related, irrelevant), and that the rationale field is present and non-empty. Reject and retry any batch where validation fails. Log validation failures with the raw model output for debugging.

Retry and Fallback Logic: Implement a retry loop with a maximum of 2 retries for validation failures. On the first retry, append the validation error message to the prompt context. On the second retry, lower the temperature to 0 and request a stricter output format. If both retries fail, route the batch to a human review queue and apply a conservative default tier (contextually-relevant) to all unclassified passages to prevent pipeline blockage. Tier Boundary Calibration: Run the prompt against a golden evaluation set of 50–100 passage-query pairs with human-assigned tier labels. Measure Cohen's kappa between model and human labels. If agreement drops below 0.7, review tier boundary definitions in the prompt and adjust the tier descriptions, not the model temperature. Recalibrate after any prompt change or model version upgrade.

Model Choice: Use a model with strong instruction-following and structured output capabilities, such as GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro. Avoid smaller models for this task unless fine-tuned on tier assignment data, as tier boundary judgment requires nuanced relevance reasoning. Logging and Observability: Log every tier assignment call with the query hash, batch ID, model version, prompt version, raw output, validation status, retry count, and final tier labels. This audit trail is essential for debugging tier drift, detecting model behavior changes, and providing evidence for compliance reviews. Human Review Integration: For high-stakes domains such as legal or medical RAG, route all directly-supporting tier assignments to a sampling-based human review queue. Reviewers should confirm that passages labeled as directly supporting the query are not misclassified contextually-relevant passages that happen to share keywords. This gate prevents overconfident evidence selection from propagating into answer generation.

Avoid These Pitfalls: Do not use the tier assignment prompt as a final relevance filter without downstream answer generation checks—a passage can be correctly tiered but still misused by the answer synthesis model. Do not skip output validation; even structured-output models occasionally drop passage IDs or produce invalid enum values under load. Do not assume tier boundaries are stable across model versions; recalibrate after every model upgrade. Finally, do not treat tier assignment as a replacement for retrieval quality monitoring—if retrieval consistently returns irrelevant passages, fix retrieval before layering on tier classification.