Irrelevant Context Flagging Prompt

This prompt is designed for RAG developers and search engineers who need to quantify noise in their retrieval pipeline by auditing production traces. Its primary job is to review every retrieved passage for a given user query and flag chunks that are irrelevant, producing a noise ratio score. You should use this prompt when you suspect that poor answer quality is caused by the retriever pulling in distracting or off-topic content, rather than by a generation or reasoning failure. It is a diagnostic tool for the retrieval step, not a fix for the generator.

The ideal workflow involves extracting a single trace from your observability platform that contains the user's original query and the full list of retrieved chunks with their content. Feed this into the prompt to get a structured audit: each chunk receives a binary relevance label, a brief justification, and an overall noise ratio is calculated. This output can be wired into a monitoring dashboard to track retrieval precision over time or used as a debugging artifact during incident response. Do not use this prompt for real-time guardrailing, as it requires a full trace and is designed for offline analysis. It is also not a substitute for evaluating the final answer's faithfulness; pair it with a grounding verification prompt for a complete picture.

Before relying on the noise ratio in production, calibrate the prompt against a set of 20–30 traces where relevance has been labeled by a human reviewer. Pay close attention to borderline cases where a chunk is topically related but does not contain a useful answer—these are the most common failure mode. If the prompt's relevance judgments consistently disagree with human labels on these edge cases, refine the [CONSTRAINTS] section with explicit examples of what counts as 'useful evidence' versus 'topical noise.' For high-stakes applications, always log the prompt's output alongside the trace for auditability and consider routing traces with a noise ratio above a defined threshold for human review.

This prompt template is designed to be copied directly into your observability or evaluation harness. It instructs the model to review each retrieved passage from a production trace, compare it against the user's original query, and flag chunks that are irrelevant despite being retrieved. The output is a structured noise ratio score and a per-chunk justification, which you can feed into dashboards, alerting thresholds, or retrieval pipeline tuning workflows.

textCopy
You are a retrieval quality auditor. Your task is to review a production trace from a RAG system and identify irrelevant context that was retrieved but does not help answer the user's query.

## INPUT

**User Query:**
[USER_QUERY]

**Retrieved Passages:**
[RETRIEVED_PASSAGES]

Each passage is formatted as:
[CHUNK_ID]: [PASSAGE_TEXT]

## TASK

For each retrieved passage, determine whether it is relevant to the user's query. A passage is relevant if it contains information that could help answer the query, provide useful background, or offer supporting evidence. A passage is irrelevant if it is off-topic, discusses unrelated subjects, or contains no information connected to the query's intent.

## OUTPUT_SCHEMA

Return a JSON object with the following structure:

{
  "total_chunks": <integer>,
  "irrelevant_chunks": <integer>,
  "noise_ratio": <float between 0.0 and 1.0>,
  "chunk_assessments": [
    {
      "chunk_id": "<string>",
      "relevant": <boolean>,
      "justification": "<one-sentence explanation of why this chunk is relevant or irrelevant>",
      "relevance_score": <integer 1-5, where 1 is completely irrelevant and 5 is highly relevant>
    }
  ],
  "summary": "<one-paragraph summary of the overall retrieval quality and noise patterns>"
}

## CONSTRAINTS

- Evaluate each chunk independently against the user query.
- Do not mark a chunk as irrelevant simply because it is a partial match; consider whether it contributes useful context.
- If a chunk is borderline, explain your reasoning in the justification field and assign a relevance_score of 3.
- The noise_ratio must equal irrelevant_chunks divided by total_chunks.
- Do not hallucinate chunk content; only reference passages provided in the input.
- If no passages are provided, return total_chunks: 0, irrelevant_chunks: 0, noise_ratio: 0.0, and an empty chunk_assessments array.

## EXAMPLES

**Example 1: Clearly Irrelevant**
User Query: "What are the symptoms of strep throat?"
Chunk: "The capital of France is Paris, a global center for art and fashion."
Assessment: relevant: false, relevance_score: 1, justification: "Discusses geography, not medical symptoms."

**Example 2: Borderline Relevance**
User Query: "How do I reset my router password?"
Chunk: "Network security best practices include changing default passwords on all devices."
Assessment: relevant: true, relevance_score: 3, justification: "Provides general security context but does not give specific reset instructions."

**Example 3: Clearly Relevant**
User Query: "What is the refund policy for annual subscriptions?"
Chunk: "Annual subscribers may request a full refund within 30 days of purchase."
Assessment: relevant: true, relevance_score: 5, justification: "Directly answers the refund policy question."

To adapt this template, replace [USER_QUERY] with the original user input from your trace and [RETRIEVED_PASSAGES] with the list of chunks returned by your retrieval pipeline. If your trace format differs, adjust the passage formatting instructions accordingly. For high-stakes applications where retrieval noise directly impacts user trust or regulatory compliance, add a human review step before accepting the noise_ratio as ground truth. Wire the output into your monitoring dashboard to track noise_ratio trends over time, and set alert thresholds when the ratio exceeds a defined SLO.

This prompt is designed to run as a post-retrieval, pre-generation audit step within your trace analysis pipeline. It should not be called in the hot path of a user-facing request. Instead, wire it into an asynchronous observability worker that consumes logged traces from your RAG system. The worker extracts the user query and the list of retrieved passages from the trace, formats them into the prompt's [USER_QUERY] and [RETRIEVED_PASSAGES] placeholders, and sends the request to a fast, cost-effective model like GPT-4o-mini or Claude 3.5 Haiku. The output is a structured JSON object containing a per-passage relevance flag and a global noise ratio score, which should be written to your monitoring database (e.g., a retrieval_quality_events table) alongside the trace ID and timestamp.

To make this production-grade, implement a strict validation layer around the model's JSON output. Use a schema validator like Pydantic or Zod to ensure the response contains the required noise_ratio float and a passages array where each item has a valid passage_id and a boolean is_irrelevant field. If validation fails, implement a single retry with a more forceful system instruction, such as 'You MUST return valid JSON matching the exact schema provided.' If the retry also fails, log the raw response as a validation_failure event for manual review and do not write a corrupted record. This prevents a bad model output from polluting your quality metrics. For high-stakes pipelines where retrieval quality directly impacts user trust, route a configurable percentage of traces (e.g., 5%) to a human review queue where an engineer can confirm borderline relevance flags before they are aggregated into dashboards.

The primary metric to track is the noise ratio over time, aggregated by knowledge base, retrieval strategy, or query category. A sudden spike in the noise ratio often indicates an upstream problem, such as a broken embedding model, a misconfigured metadata filter, or a corrupted search index. Wire the noise ratio into your existing observability stack (e.g., Datadog, Grafana) as a time-series metric, and set an alert threshold based on historical baselines. For example, if your typical noise ratio is 0.15, an alert at 0.40 can catch retrieval degradation before it causes user-facing hallucinations. Pair this prompt with the 'Retrieved Chunk Relevance Scoring Prompt' for a deeper diagnostic view when the noise ratio crosses the alert threshold.