Adaptive Noise Threshold Prompt Based on Query Complexity

This prompt is for RAG pipeline operators and AI engineers who need to dynamically adjust noise-filtering strictness based on query complexity. The job-to-be-done is preventing simple factoid queries from drowning in irrelevant retrieved chunks while ensuring complex, multi-aspect questions don't lose critical evidence to over-aggressive filtering. You should use this prompt when your retrieval system returns a variable-quality context set and a static filter threshold either lets too much noise through for simple queries or strips essential context from difficult ones.

The ideal user has a production RAG system with observable retrieval quality metrics and can classify query complexity either heuristically (query length, entity count) or via a lightweight upstream classifier. Required context includes the raw query string, the retrieved passage set with relevance scores, and a defined complexity taxonomy—typically simple factoid, moderate multi-constraint, and complex multi-hop or comparative. Do not use this prompt when your retrieval pipeline already produces consistently clean results, when query complexity is uniform, or when latency constraints prevent an additional classification step before filtering.

Before deploying, establish baseline noise ratios for each complexity tier so you can validate that the adaptive thresholds actually improve answer quality rather than adding unnecessary system complexity. The prompt outputs a threshold decision with complexity rationale, making it auditable. Pair this with a context quality gate evaluation downstream to catch cases where the adaptive threshold was too aggressive. If your domain is regulated, log the complexity classification and threshold choice alongside the final answer for review.

This prompt template implements an adaptive noise filter for RAG pipelines. Instead of applying a single static relevance threshold to all queries, it evaluates the incoming query's complexity and selects a filtering strictness level accordingly. Simple factoid queries get tight filters to exclude marginal passages; complex multi-aspect or comparative queries get looser filters to preserve potentially useful context that might otherwise be pruned. The prompt expects a query and a set of retrieved passages, and it returns a structured decision containing the complexity classification, the recommended threshold, and the reasoning behind both.

textCopy
You are a retrieval quality controller for a RAG system. Your job is to analyze a user query and determine the appropriate noise-filtering strictness for the retrieved context.

## INPUT

Query: [QUERY]

Retrieved Passages (each with ID and content):
[RETRIEVED_PASSAGES]

## TASK

1. Classify the query complexity into one of these categories:
   - "simple_factoid": A single, well-defined fact or entity lookup (e.g., "What is the capital of France?", "When was Python 3.12 released?").
   - "moderate": A question requiring one or two pieces of evidence with minor synthesis (e.g., "What are the main differences between Redis and Memcached?", "Summarize the side effects of drug X").
   - "complex_multi_aspect": A question requiring multiple pieces of evidence, comparison across sources, temporal reasoning, or multi-hop synthesis (e.g., "How has the company's revenue strategy evolved over the last three fiscal years and what risks remain?").

2. Based on the complexity classification, recommend a noise-filtering threshold:
   - For "simple_factoid": Use a STRICT threshold. Only passages directly answering the fact should pass. Discard tangential, background, or loosely related content.
   - For "moderate": Use a MODERATE threshold. Keep passages with clear relevance and those providing useful context. Discard clearly off-topic or boilerplate content.
   - For "complex_multi_aspect": Use a LOOSE threshold. Keep passages with partial relevance, background context, or adjacent topics that might contribute to a complete answer. Only discard clearly irrelevant or distractor content.

3. Provide a brief rationale explaining:
   - Why the query fits the chosen complexity class.
   - Why the recommended threshold is appropriate for that class.
   - Any risks of over-filtering or under-filtering given the query type.

## OUTPUT_SCHEMA

Return a JSON object with this exact structure:
{
  "query_complexity": "simple_factoid" | "moderate" | "complex_multi_aspect",
  "recommended_threshold": "strict" | "moderate" | "loose",
  "rationale": "string explaining the classification and threshold choice",
  "over_filtering_risk": "low" | "medium" | "high",
  "under_filtering_risk": "low" | "medium" | "high"
}

## CONSTRAINTS

- Do not answer the query. Only classify it and recommend a threshold.
- Base the complexity classification on the query structure, not on the retrieved passages.
- If the query is ambiguous, default to "moderate" complexity and note the ambiguity in the rationale.
- The over_filtering_risk and under_filtering_risk fields must reflect the consequences of applying the recommended threshold to this specific query type.

To adapt this template, replace [QUERY] with the user's raw question and [RETRIEVED_PASSAGES] with the full set of retrieved chunks, each labeled with an ID. The output schema is designed to feed directly into a downstream filtering step: use the recommended_threshold field to select a relevance score cutoff, and log the rationale and risk fields for observability. For production deployments, validate that the returned JSON matches the schema before acting on the threshold decision. If the model returns an invalid complexity class or threshold value, fall back to a "moderate" threshold and log the failure for review. This prompt works best with models that have strong instruction-following and JSON output capabilities; test across your model candidates to confirm schema compliance under load.

The adaptive noise threshold prompt is not a standalone module—it is a decision point inside a context assembly pipeline. Wire it after retrieval and before answer generation. The prompt receives a query, a complexity classification (or the raw query for self-classification), and the retrieved context set. It returns a threshold decision, a complexity rationale, and a filtered context list. The application layer must parse this structured output and apply the threshold to downstream context pruning or reranking steps. Do not pass the raw LLM output directly to the answer generator without validating the threshold value and confirming that the filtered context is non-empty.

Implement this as a dedicated pipeline stage with strict input/output contracts. The input contract requires: [QUERY] (the original user question), [RETRIEVED_CONTEXT] (an array of passage objects with id, text, and optional score fields), and [COMPLEXITY_HINT] (optional pre-classification from a faster classifier or set to null for self-classification). The output contract expects a JSON object with complexity_level (enum: simple, moderate, complex), complexity_rationale (string), noise_threshold (float between 0.0 and 1.0), and filtered_context (array of passage objects that meet the threshold). Add a JSON schema validator after the LLM call. If parsing fails, retry once with the error message injected into the prompt. If the retry also fails, log the failure, fall back to a conservative threshold of 0.5, and flag the request for review. For high-stakes domains, route threshold decisions below 0.3 or above 0.9 to a human approval queue before context pruning executes.

Model choice matters here. This prompt requires strong instruction-following and structured output reliability. Use GPT-4o, Claude 3.5 Sonnet, or an equivalent model with JSON mode enabled. Avoid smaller models for this stage—threshold miscalibration cascades into either over-filtering (losing critical evidence) or under-filtering (passing noise to the answer generator). For latency-sensitive pipelines, pre-classify query complexity with a fast, cheap model (e.g., a fine-tuned classifier or a lightweight LLM call) and pass the result as [COMPLEXITY_HINT] to skip self-classification. Log every threshold decision with the query, complexity level, threshold value, and pre/post-filter context counts. These logs become essential for tuning threshold boundaries and detecting drift in retrieval quality over time.

Testing this prompt requires a labeled eval set of queries with known complexity levels and ground-truth relevant passages. Measure two failure modes: over-filtering (threshold too strict, removing passages needed to answer the query) and under-filtering (threshold too loose, retaining noise that degrades answer quality). Run the prompt against your eval set and compute precision/recall of retained relevant passages at each complexity level. If simple queries show under-filtering, tighten the threshold mapping in the prompt instructions. If complex queries show over-filtering, loosen it. Automate these evals in CI before deploying prompt changes. The next step is to integrate the filtered context output into your answer generation stage, ensuring the answer prompt receives only the passages that passed the adaptive threshold.