Retrieval Noise Classification Prompt Template

This prompt is for RAG operators and search engineers who need to monitor retrieval quality at scale. Use it when you have a set of retrieved passages for a query and need to know how much noise entered the context window before answer generation. The prompt classifies each passage as relevant, partially relevant, off-topic, or distractor noise, then computes aggregate noise ratios. This is a diagnostic prompt for pipeline observability, not a filtering step. Run it on sampled queries to track retrieval drift, evaluate embedding model changes, or set alert thresholds on noise levels.

Do not use this prompt when you need to remove noise inline during a user-facing request; use a filtering prompt for that. This prompt assumes you already have retrieved passages with source identifiers and a clear user query. The output is a structured JSON report containing per-passage labels with reasoning and aggregate metrics like the noise-to-signal ratio. Wire this into your eval harness by sampling production queries, running the prompt on their retrieval results, and logging the noise ratio to your observability stack. Set a threshold alert when the distractor noise ratio exceeds your acceptable baseline—typically 20-30% for high-quality retrieval pipelines, though your tolerance will vary by domain.

Before deploying, validate the prompt against a golden dataset of pre-labeled passages to confirm classification accuracy. Common failure modes include the model conflating partially relevant passages with noise, or over-flagging domain-specific terminology as off-topic. If your retrieval pipeline serves regulated or high-stakes domains, always include a human review step for the first few hundred classifications to calibrate thresholds. Once calibrated, this prompt becomes a reliable canary for retrieval degradation—run it after any embedding model update, chunking strategy change, or vector store migration.

This prompt is the core instruction set for a retrieval noise classifier. It takes a user query and a set of retrieved passages, then labels each passage as relevant, partially_relevant, off_topic, or distractor. The output is a structured JSON object containing the labeled passages and a computed noise_ratio metric. You can paste this directly into your evaluation harness, an LLM playground, or your RAG pipeline's pre-processing step. Replace every square-bracket placeholder with your actual data before sending it to the model.

textCopy
You are a retrieval quality classifier. Your task is to evaluate each retrieved passage against the user query and classify it into exactly one of four categories.

## INPUT
**User Query:** [USER_QUERY]
**Retrieved Passages:** [RETRIEVED_PASSAGES]

## CLASSIFICATION RULES
For each passage, assign one label:
- `relevant`: The passage directly answers or provides substantial evidence for the query.
- `partially_relevant`: The passage touches on the query's domain or entities but does not directly address the core question.
- `off_topic`: The passage is unrelated to the query's intent, despite possible keyword overlap.
- `distractor`: The passage appears relevant at first glance but contains information that would actively mislead or contradict a correct answer.

## CONSTRAINTS
- Do not fabricate labels. If you are uncertain, default to `off_topic` and note your uncertainty in the `classification_notes` field.
- Evaluate each passage independently. Do not let one passage's label influence another's.
- If a passage is empty or contains only boilerplate (e.g., navigation menus, copyright footers), classify it as `off_topic`.

## OUTPUT SCHEMA
Return a single JSON object with this exact structure:
{
  "classifications": [
    {
      "passage_id": "string",
      "label": "relevant | partially_relevant | off_topic | distractor",
      "classification_notes": "Brief explanation of why this label was chosen."
    }
  ],
  "noise_ratio": "float between 0.0 and 1.0, calculated as (off_topic + distractor) / total_passages",
  "summary": "One-sentence assessment of overall retrieval quality."
}

## EXAMPLES
Query: "What is the return policy for online orders?"
Passage: "Customers can return online purchases within 30 days for a full refund. Items must be unworn with tags attached."
Label: relevant
Notes: "Directly states the return window and conditions for online orders."

Query: "What is the return policy for online orders?"
Passage: "Our store hours are Monday through Saturday, 9 AM to 9 PM."
Label: off_topic
Notes: "Discusses store hours, not return policies."

Query: "What is the return policy for online orders?"
Passage: "All sales are final. We do not accept returns under any circumstances."
Label: distractor
Notes: "Appears to be a return policy but contradicts the company's actual 30-day return policy and would mislead the answer if used."

After copying the template, adapt the [USER_QUERY] and [RETRIEVED_PASSAGES] placeholders with your live data. The [RETRIEVED_PASSAGES] should be a JSON array of objects, each with at least an id and text field. If your retrieval system returns metadata like source URLs or scores, include those in the passage objects—the model can use them to improve classification accuracy. For high-stakes pipelines, add a [RISK_LEVEL] placeholder and a corresponding instruction to escalate distractor passages for human review before they reach downstream answer generation. Test this prompt with a golden dataset of 20–50 query-passage pairs where you already know the correct labels, and measure precision and recall per class before deploying.

This prompt is an observability tool, not a runtime answer generator. It should run asynchronously on a sample of retrieval results—never in the critical path of a user request. The goal is to produce labeled context sets and noise ratio metrics that feed into dashboards, alerting thresholds, and retrieval pipeline improvement cycles. Wire it into a background evaluation harness that pulls query–retrieved passage pairs from production logs, runs classification, and writes structured results to a metrics store.

Build the harness to handle batching, retries, and schema validation. For each query, send the retrieved passages alongside the query text and any metadata filters used. Expect the model to return a JSON array of passage labels with passage_id, relevance_label (one of relevant, partially_relevant, off_topic, distractor), and a noise_ratio summary. Validate the output against this schema before ingestion. If validation fails, retry once with a stricter prompt that includes the schema inline. Log all failures and schema violations for later prompt debugging. Use a model with strong JSON mode support—GPT-4o or Claude 3.5 Sonnet are good defaults; avoid smaller models that may drift on the label taxonomy.

Store results with query timestamp, retrieval configuration, and model version so you can track noise drift over time. Set alert thresholds on the noise ratio metric: if the proportion of off_topic plus distractor passages exceeds a defined limit (e.g., 40%) for a sustained period, trigger a retrieval quality review. Do not use this prompt to auto-drop passages in production until you have calibrated the classifier against human-labeled data and confirmed low false-positive rates on relevant passages. The next step is to run a calibration pass with human review on a few hundred samples before trusting the noise ratio for automated decisions.