Query-Focused Context Filtering Prompt

This prompt is for retrieval pipeline developers who need to clean a noisy context window before answer generation. The job is to evaluate each retrieved passage against the user's actual query intent—not just keyword overlap—and filter out passages that match on surface terms but miss the semantic need. The ideal user is an AI engineer or RAG system builder who already has a retrieval step producing candidate passages and needs a reliable filtering layer that reduces hallucination risk and token waste before the answer synthesis step.

Use this prompt when your retrieval system returns passages that contain the right keywords but the wrong meaning—for example, a query about 'Python packaging' returning documentation about 'Python snake handling.' It is also appropriate when you need per-passage alignment scores for monitoring, threshold tuning, or downstream confidence-weighted answer generation. The prompt expects a structured input containing the original query and a list of candidate passages with source identifiers. It outputs filtered passages with query-alignment scores and explicit removal reasons, making the filtering decision auditable.

Do not use this prompt as a replacement for retrieval quality improvements. If your vector search or keyword retrieval is fundamentally broken, fix retrieval before adding a filtering layer. Do not use this prompt when the context window is already small and clean—the filtering overhead isn't worth it for under 5 passages. Do not use this prompt for real-time streaming applications where the added latency of an LLM filtering pass would violate latency budgets. For high-stakes domains like healthcare or legal, always pair this prompt with human review of the filtered context set and run an over-filtering risk assessment to verify that answer-critical evidence wasn't removed. The next section provides the copy-ready template you can adapt and wire into your pipeline.

This prompt template is designed to be dropped into a RAG pipeline after retrieval but before answer generation. It takes a user query and a set of retrieved passages, then evaluates each passage for genuine query alignment—not just keyword overlap. The output is a filtered context set with alignment scores, making it easy to set thresholds, log decisions, and debug retrieval quality in production.

codeCopy
You are a context filtering engine. Your job is to evaluate each retrieved passage against the user's query intent and filter out passages that match keywords but miss the semantic need.

## INPUT

**User Query:** [QUERY]

**Retrieved Passages:**
[PASSAGES]

## INSTRUCTIONS

For each passage, determine whether it genuinely addresses the user's query intent. A passage should be RETAINED only if it provides information that helps answer the question. A passage should be FILTERED OUT if it:
- Contains matching keywords but addresses a different topic or intent
- Discusses the same domain but answers a different question
- Is tangentially related but provides no actionable information for this query
- Contains only background, boilerplate, or navigational content

## OUTPUT SCHEMA

Return a JSON object with the following structure:

{
  "query_intent": "Brief description of what the user is actually asking for",
  "filtered_passages": [
    {
      "passage_id": "original index or identifier from input",
      "text": "original passage text",
      "decision": "RETAINED",
      "alignment_score": 0.0 to 1.0,
      "rationale": "Why this passage aligns with the query intent"
    }
  ],
  "removed_passages": [
    {
      "passage_id": "original index or identifier",
      "text": "original passage text",
      "decision": "FILTERED_OUT",
      "alignment_score": 0.0 to 1.0,
      "rationale": "Why this passage was removed despite any keyword matches"
    }
  ],
  "filtering_summary": {
    "total_input_passages": number,
    "retained_count": number,
    "removed_count": number,
    "average_alignment_score_retained": number
  }
}

## CONSTRAINTS

[CONSTRAINTS]

## EXAMPLES

[EXAMPLES]

To adapt this template, replace [QUERY] with the user's original question and [PASSAGES] with your retrieved context set—typically a JSON array of objects with passage_id and text fields. Use [CONSTRAINTS] to add domain-specific rules, such as 'Prefer passages from authoritative sources' or 'Require alignment_score >= 0.7 for retention.' Use [EXAMPLES] to provide 2-3 few-shot demonstrations showing borderline cases where keyword matches should be rejected. For production, validate the output JSON against the schema before passing filtered passages to your answer generation step. If the retained count is zero, route to a fallback that either expands retrieval or abstains from answering.

This prompt operates as a pre-answer filtering stage inside a RAG pipeline. It receives a batch of retrieved passages and a user query, then returns a filtered subset with query-alignment scores. The output is typically consumed by a downstream answer-generation prompt. The harness should treat this filter as a gating step: if the filtered context set is empty or all scores fall below a configured threshold, the pipeline should route to an abstention or clarification path rather than forcing an answer from weak evidence.

Integration pattern: Place this prompt after retrieval and before answer synthesis. The typical call flow is: (1) retrieve top-k passages from your vector store or search index, (2) assemble the prompt with [QUERY] and [PASSAGES] placeholders populated, (3) call the model with response_format set to the expected JSON schema, (4) validate the output structure, (5) apply a minimum query_alignment_score threshold (start with 0.6 and tune on eval data), and (6) pass surviving passages to the answer-generation prompt. For high-throughput systems, batch multiple queries in a single call if your model supports it, but keep per-batch passage counts under the model's context window minus prompt overhead.

Validation and retry logic: Parse the model's JSON output immediately. Check that filtered_passages is an array, each entry has passage_id, query_alignment_score, and retention_decision fields, and scores are floats between 0.0 and 1.0. If validation fails, retry once with the same prompt plus the validation error message appended as a [REPAIR_INSTRUCTION]. If the retry also fails, log the raw output and fall back to passing all passages through unfiltered with a warning flag. For high-stakes domains, route validation failures to a human review queue rather than silently degrading.

Model choice and latency: This is a classification-and-filtering task, not a generation task. Smaller, faster models (e.g., Claude Haiku, GPT-4o-mini, or fine-tuned open-weight models) often perform well here. Measure latency at your expected passage volume; if filtering adds more than 500ms at p95, consider reducing the number of passages sent per call or using a streaming parse approach. Cache filtering results by (query_hash, passage_id) if identical queries repeat in your workload.

Observability: Log the input passage count, output passage count, score distribution, and any validation failures. Emit metrics for filter rate (percentage of passages removed), mean alignment score, and empty-result rate. These metrics are your primary signals for threshold tuning and drift detection. If the filter rate suddenly spikes or drops, investigate retrieval quality changes before adjusting the prompt.

What to avoid: Do not use this prompt as the sole quality gate without an over-filtering check. After filtering, run a lightweight coverage check—either a second LLM call or a heuristic—to verify that answer-critical entities or facts weren't removed. Do not skip the abstention path when all passages are filtered out; an empty context set should trigger a "not enough information" response, not a hallucinated answer. Do not hardcode the alignment threshold without making it configurable per use case or query type.