Top-K Passage Selection Prompt for Constrained Context Windows

This prompt is designed for retrieval-augmented generation (RAG) pipelines where the total retrieved evidence exceeds the model's context window or a strict token budget. Instead of naive truncation, which discards potentially critical information, this prompt instructs the model to select an optimal subset of passages that maximizes information coverage, relevance, and diversity. Use this when you need a smart, budget-aware gate between retrieval and answer synthesis. It is ideal for production systems where cost and latency constraints require packing the most signal into a limited context.

The ideal user is an AI engineer or search engineer who has already tuned their retrieval step but now faces a context-packing problem. You have more good passages than you can fit. You need a selection step that understands query intent, avoids redundant information, and preserves complementary facts from different sources. This prompt works best when your upstream retrieval returns 20–100 passages and your downstream synthesis model has a budget of 4K–16K tokens. It is not a replacement for retrieval quality—if your top-10 retrieval is already poor, no selection prompt will fix it. It also assumes passages are already chunked and have basic metadata like source titles or IDs.

Do not use this prompt when you have only a handful of passages that easily fit in context, when you need real-time sub-50ms decisions, or when your passages are extremely long and cannot be individually evaluated. In those cases, consider embedding-based re-ranking, simple recency filters, or expanding your context window. For high-stakes domains like healthcare or legal, always log the selected and excluded passages for auditability, and consider a human review step before answers reach users. After implementing this prompt, pair it with an evidence sufficiency check to confirm the selected subset actually contains enough information to answer the question.

This prompt template is designed to be placed directly into your system or user prompt, acting as a deterministic filter before answer synthesis. It forces the model to make explicit trade-offs between relevance, information diversity, and token cost. The core logic asks the model to prioritize passages that are not only highly relevant but also complementary, ensuring that the final selection covers distinct aspects of the user's query rather than repeating the same information from multiple sources. You should use this when your retrieval step returns more passages than can fit in a single context window and you need a programmatic, auditable selection step.

textCopy
You are an evidence selection specialist. Your task is to select the optimal subset of passages from the provided [RETRIEVED_PASSAGES] that maximizes information coverage for the [USER_QUERY] while strictly adhering to a token budget of [MAX_TOKENS].

# Selection Criteria
1.  **Relevance:** The passage must directly address the [USER_QUERY] or a necessary sub-component of it.
2.  **Information Density:** Prefer passages with high factual content per token. Avoid verbose, redundant, or preamble-heavy passages.
3.  **Complementarity:** Select passages that provide *distinct* pieces of evidence. If two passages convey the same fact, choose the one with higher source credibility or recency and discard the other.
4.  **Source Credibility:** Prioritize passages from sources with higher [CREDIBILITY_WEIGHT] based on authority, recency, and domain expertise.

# Constraints
- The total token count of all selected passages MUST NOT exceed [MAX_TOKENS].
- You must select at least [MIN_PASSAGES] passages, unless fewer than [MIN_PASSAGES] are provided or relevant.
- Do not modify the text of the selected passages. Output them exactly as provided.
- If no passages are sufficiently relevant, return an empty selection and set `sufficient_evidence` to `false`.

# Output Format
You must respond with a valid JSON object conforming to this exact schema:
{
  "sufficient_evidence": boolean,
  "total_selected_tokens": number,
  "selection_rationale": "string explaining the trade-offs made",
  "selected_passages": [
    {
      "passage_id": "string",
      "passage_text": "string",
      "selection_reason": "string"
    }
  ],
  "excluded_passages": [
    {
      "passage_id": "string",
      "exclusion_reason": "string"
    }
  ]
}

# Input Data
**User Query:** [USER_QUERY]
**Token Budget:** [MAX_TOKENS]
**Retrieved Passages:** [RETRIEVED_PASSAGES]

To adapt this template, start by replacing the square-bracket placeholders with your application's dynamic variables. The [RETRIEVED_PASSAGES] should be a pre-formatted list, likely as a JSON array of objects each containing an id, text, and optional metadata like a date or source authority score. The [CREDIBILITY_WEIGHT] placeholder can be replaced with a static instruction like 'prioritize peer-reviewed sources and official documentation' or a dynamic variable if you have a pre-calculated score. The most critical adaptation is the [MAX_TOKENS] value, which you must calculate dynamically based on your model's context window minus the tokens reserved for the system prompt, user query, and the expected output tokens for the final answer. A common failure mode is setting this budget too high, leaving no room for the synthesis step, so implement a token-counting function in your application harness to set this value precisely before calling the model.

The Top-K Passage Selection prompt is designed to sit between your retrieval step and your answer synthesis step. It receives a ranked or unranked set of candidate passages and a strict token budget, then returns the optimal subset. In production, this prompt is typically called inside a pre-synthesis filtering function that enforces the budget before the final LLM call. The harness must handle three critical concerns: budget enforcement, output validation, and overflow fallback.

Budget enforcement should happen at the application layer, not just in the prompt. Before calling the selection prompt, calculate the available token budget by subtracting your system prompt, synthesis prompt, and output reservation from the model's context window. Pass this budget as [TOKEN_BUDGET]. After receiving the selected passages, re-count tokens using the same tokenizer your model uses (e.g., tiktoken for OpenAI models). If the returned set exceeds the budget, apply a hard truncation that drops the lowest-ranked passage first, then re-check. Log every budget violation as a warning so you can tune the prompt's compliance rate over time. For high-throughput systems, consider setting the prompt's budget 5-10% below the actual limit to create a safety margin.

Output validation must verify the JSON schema before the selected passages reach synthesis. Check that selected_passages is a valid array, each entry has a passage_id and text field, no passage appears twice, and the exclusion_reasons map references only passages that were excluded. If validation fails, do not silently proceed with partial data. Implement a retry with stricter instructions: append the specific validation error to the next attempt and reduce the temperature. After two retries, fall back to a simpler selection method (e.g., top-N by retrieval score) and log the incident for review. For regulated domains, flag validation failures for human review before any answer reaches the user.

Overflow handling is the most common production failure mode. When the candidate set is so large or passages are so long that even the selection prompt plus the minimum viable context exceeds the model's window, you need a pre-filter. Implement a two-stage selection: first, use a lightweight relevance score threshold or a smaller, faster model to reduce candidates to 2-3x the target count. Only then invoke the full Top-K selection prompt. If the prompt itself fails due to context length, catch the error, apply aggressive truncation to the input passages (keeping only title and first 200 tokens of each), and retry once. If that also fails, escalate to a human operator or return a controlled degradation response indicating insufficient context.

Model choice matters for this prompt. The selection task requires comparative reasoning across passages, not just per-passage scoring. Use a model with strong instruction-following and JSON output capabilities (e.g., GPT-4o, Claude 3.5 Sonnet, or a fine-tuned variant if you have consistent selection patterns). Avoid using the cheapest or fastest model here unless you've validated that its selection quality doesn't degrade your downstream answer accuracy. For cost-sensitive pipelines, consider caching selection results when the same query and candidate set reappear, using a hash of the normalized input as the cache key. Log every selection decision with the query, candidate IDs, selected IDs, exclusion reasons, token counts, and latency. This trace data is essential for debugging hallucination incidents and tuning your retrieval pipeline.