Query Variant Scoring and Selection Prompt for Retrieval Pipelines

This prompt is for infrastructure teams that have already generated multiple candidate query variants—from dense vector rewrites, sparse keyword expansions, graph traversals, or other backends—and now need to decide which subset to execute. The job-to-be-done is scoring and selection: given a set of candidate queries, a retrieval objective, and backend constraints, the prompt evaluates each variant, assigns a score, and produces a ranked, justified selection set. The ideal user is a retrieval pipeline engineer or RAG architect who understands their backend capabilities and can define what 'good' looks like for their system. Required context includes the original user question, the list of candidate query variants, the available retrieval backends and their strengths, and the evaluation criteria (e.g., recall priority, precision priority, latency budget, or cost constraints).

Do not use this prompt when you need to generate query variants—that belongs upstream in the pipeline with dedicated expansion, decomposition, or backend-specific rewriting prompts. This prompt assumes candidates already exist. It is also not a replacement for retrieval result evaluation; it scores queries before execution, not after. Avoid using it when the selection criteria are purely deterministic (e.g., 'always run all variants' or 'pick the first three')—a simple rule in application code is cheaper and more reliable. The prompt adds value when selection requires semantic judgment: weighing trade-offs between recall and precision, detecting redundant variants, or justifying why a particular formulation is better suited to a specific backend.

Before wiring this into production, define your evaluation rubric explicitly. The prompt needs concrete scoring dimensions—such as semantic alignment with user intent, backend suitability, term specificity, and variant diversity—not vague instructions like 'pick the best ones.' Build a small golden dataset of query variant sets with expected selections and use it for regression testing whenever the prompt or scoring criteria change. If the selection decision gates expensive retrieval calls or affects downstream answer quality, add a human review step for low-confidence selections or flag cases where score distributions are flat (indicating no clear winner). The next section provides the copy-ready template you can adapt with your own scoring dimensions and backend descriptions.

This prompt template evaluates a set of generated query variants against your defined retrieval objectives and selects the optimal subset for execution. It is designed to sit between query generation and retrieval dispatch in a multi-stage pipeline, ensuring that only high-quality, complementary variants are sent to backends. The template requires you to supply the original user query, the candidate variants, and your retrieval objectives. It returns a scored, ranked, and justified selection list that your application can parse and act on.

textCopy
You are a retrieval pipeline quality controller. Your task is to score and select query variants for execution against a set of retrieval objectives.

## INPUT
- Original User Query: [ORIGINAL_QUERY]
- Candidate Query Variants: [CANDIDATE_VARIANTS]
- Retrieval Objectives: [RETRIEVAL_OBJECTIVES]
- Backend Constraints: [BACKEND_CONSTRAINTS]
- Selection Budget: [MAX_VARIANTS]

## OUTPUT SCHEMA
Return a JSON object with this exact structure:
{
  "selected_variants": [
    {
      "variant": "string",
      "score": number,
      "justification": "string",
      "backend_assignment": "string"
    }
  ],
  "rejected_variants": [
    {
      "variant": "string",
      "rejection_reason": "string"
    }
  ],
  "selection_summary": "string"
}

## SCORING RULES
1. Score each variant from 0.0 to 1.0 against EVERY retrieval objective.
2. A variant's final score is the weighted average across objectives.
3. Penalize variants that are near-duplicates of higher-scored variants (cosine similarity > 0.85).
4. Prefer variants that cover different retrieval strategies (dense, sparse, graph) when objectives require diversity.
5. Reject variants that violate backend constraints or are semantically contradictory to the original query.

## CONSTRAINTS
- Do not select more than [MAX_VARIANTS] variants.
- Every selected variant must have a backend assignment matching [BACKEND_CONSTRAINTS].
- Justifications must reference specific retrieval objectives.
- If no variant scores above 0.3, select none and explain why in the summary.

## EVALUATION CHECK
Before returning, verify:
- Scores are consistent with the stated objectives.
- No duplicate or near-duplicate variants are selected.
- The selection summary accurately reflects the trade-offs made.

To adapt this template, replace each square-bracket placeholder with concrete values from your pipeline. [ORIGINAL_QUERY] should be the raw user input. [CANDIDATE_VARIANTS] is the list of query strings produced by your upstream generation step. [RETRIEVAL_OBJECTIVES] should enumerate what matters for your use case—such as recall, precision, latency, diversity, or backend-specific constraints. [BACKEND_CONSTRAINTS] specifies which backends are available and their requirements. [MAX_VARIANTS] caps the number of variants sent downstream to control cost and latency. If your pipeline does not use weighted objectives, simplify the scoring rules to a single criterion. If you need deterministic selection, add a tie-breaking rule such as preferring the shortest variant or the one with the highest individual objective score.

This prompt operates as a scoring and selection gate between query variant generation and retrieval execution. It receives a batch of candidate query variants—typically produced by upstream rewriters, expanders, or decomposition prompts—and must return a ranked, justified selection of which variants to execute against which backends. The harness should treat this as a deterministic step in a retrieval DAG: the prompt's output directly controls which indexes receive traffic, so validation failures must halt execution rather than silently falling through to retrieval with unvetted queries.

Integration pattern: Place this prompt after all variant generation steps and before any retrieval dispatch. The input payload should include the original user query, the generated variants with their source metadata (which rewriter produced each, intended backend type, confidence scores if available), and the retrieval objectives (e.g., maximize recall, balance precision, optimize for dense-only). The output schema must be strictly validated before proceeding: check that every selected variant has a non-empty justification, that the selected boolean field is present, and that at least one variant is selected unless the prompt explicitly returns an empty selection with a documented reason. Use a JSON Schema validator in your application layer—do not rely on the model to always produce valid JSON. On validation failure, retry once with the error message injected into the prompt context. If the retry also fails, log the full payload and escalate to a human review queue rather than silently proceeding.

Model choice and latency budget: This is a classification-and-ranking task with structured output requirements. Use a model with strong instruction-following and JSON mode support (GPT-4o, Claude 3.5 Sonnet, or equivalent). Set temperature=0 to maximize scoring consistency across calls. Expect 200-800ms latency for typical variant batches of 5-15 candidates. If your retrieval pipeline has a sub-200ms total budget, consider pre-computing variant selections for common query templates or caching selections keyed by query embedding clusters. Logging and observability: Log the full prompt input, output, validation status, and selected variant IDs to your tracing system. Tag spans with prompt_id, variant_count, selection_count, and validation_passed. This data is critical for debugging retrieval quality regressions—when recall drops, you need to know whether the scoring prompt excluded a variant that would have retrieved the answer. Failure modes to monitor: (1) The prompt selects all variants with weak justifications, indicating it's not actually discriminating; (2) it consistently rejects variants from a specific rewriter, suggesting a bias in the scoring criteria; (3) justifications contradict the stated retrieval objectives; (4) the output omits the justification field entirely. Set up eval assertions that spot-check justification quality against human judgments on a golden set of 50 query-variant pairs.

Next steps after implementation: Once the harness is in place, run a shadow evaluation where you execute all variants regardless of the scoring prompt's selection, then compare retrieval quality between the selected subset and the full set. This measures the prompt's selection precision—you want high recall with fewer queries. If the selected subset underperforms, adjust the scoring criteria in the prompt template or add few-shot examples showing correct selection behavior for your domain. Avoid the temptation to bypass this prompt when retrieval results look acceptable; the selection step exists to reduce wasted compute and prevent noisy results from poorly-matched variants, and its value compounds as your variant generation surface grows.