Confidence Scoring Prompt for Query Correction Candidates

This prompt is for retrieval pipeline operators who generate multiple correction hypotheses for a single user query and need to rank them before committing to a retrieval round. The core job-to-be-done is calibrated candidate selection: given a raw user query and a list of proposed corrections, return a ranked list with per-candidate confidence scores and explicit rationale so the downstream orchestrator can decide which corrections to execute, which to discard, and whether to fall back to the original query. The ideal user is a search engineer or RAG architect who already has a correction generation step in place—spelling fixes, acronym expansion, jargon translation—and now needs a scoring layer to avoid executing low-quality or intent-breaking corrections.

Use this prompt when you have multiple correction candidates and need to choose among them. It is appropriate when the cost of executing a bad correction is high—wasted vector search compute, irrelevant retrieved context, or a broken answer downstream. The prompt requires a well-formed input structure: the original query, a list of candidate corrections, and optionally a domain glossary or context snippet to ground the scoring. Do not use this prompt as a standalone correction generator; it scores candidates, it does not produce them. Do not use it when you have only one candidate and no ranking decision to make—a simpler validation check suffices. Avoid it in ultra-low-latency paths where the scoring step adds unacceptable overhead; in those cases, pre-compute confidence heuristics or use a smaller model for the scoring pass.

Before wiring this into production, prepare a held-out ground truth set of query-correction pairs with known-good and known-bad corrections so you can calibrate the confidence scores. The prompt outputs a ranked list, but the real test is whether the top-ranked correction matches your ground truth and whether the confidence scores correlate with actual retrieval quality improvements. If the scoring model consistently assigns high confidence to corrections that degrade retrieval, the prompt needs tuning—likely through better few-shot examples that illustrate intent-breaking corrections or through tighter constraints on what constitutes a high-confidence correction. Start with a small calibration run against 50–100 annotated examples before scaling to production traffic.

This template is designed to be dropped directly into your retrieval pipeline's pre-processing stage. It expects a list of correction hypotheses generated by an upstream spell-check, acronym expansion, or normalization step, and it returns a ranked list with confidence scores. The prompt uses square-bracket placeholders for all dynamic inputs, making it safe to parameterize in code before sending to the model.

textCopy
You are a query correction evaluator for a retrieval system. Your job is to score multiple candidate corrections for a user's original query and rank them by how likely they are to preserve the user's intent while fixing errors.

**Original User Query:**
[ORIGINAL_QUERY]

**Correction Candidates:**
[CORRECTION_CANDIDATES]

**Domain Context:**
[DOMAIN_CONTEXT]

**Evaluation Criteria:**
[EVALUATION_CRITERIA]

**Output Schema:**
Return a JSON object with a "ranked_candidates" array. Each element must contain:
- "rank": integer starting at 1
- "corrected_query": string
- "confidence_score": float between 0.0 and 1.0
- "rationale": string explaining the score
- "risk_flags": array of strings (empty if none)

**Constraints:**
[CONSTRAINTS]

**Examples:**
[EXAMPLES]

**Risk Level:** [RISK_LEVEL]

Return only the JSON object. Do not include any other text.

To adapt this template, replace each bracketed placeholder with concrete values at runtime. [ORIGINAL_QUERY] is the raw user input. [CORRECTION_CANDIDATES] should be a structured list of candidate strings from your upstream correction step. [DOMAIN_CONTEXT] is critical for disambiguation—include a glossary of accepted terms, known acronyms, and proper nouns that should not be corrected. [EVALUATION_CRITERIA] should specify what makes a good correction for your use case, such as 'prefer minimal edits,' 'preserve technical jargon,' or 'penalize changes to named entities.' [CONSTRAINTS] can enforce rules like 'never change product codes matching regex [A-Z]{2}-\d{4}.' [EXAMPLES] should include 2-3 few-shot demonstrations of original queries, candidates, and the desired scoring output. [RISK_LEVEL] should be set to high for regulated domains, which should trigger downstream human review of low-confidence or flagged corrections.

The confidence scoring prompt is designed to sit between a correction candidate generator and the retrieval execution layer. In a typical pipeline, a spelling corrector or acronym expander produces multiple hypotheses for a malformed user query. This prompt receives those candidates and ranks them, assigning a confidence score and rationale to each. The output is a structured JSON list that downstream orchestration code can consume to decide which query to execute, whether to execute multiple queries in parallel, or whether to escalate for human review. The prompt is not a standalone correction tool—it assumes candidates already exist and focuses exclusively on scoring and ranking them.

Integration pattern: Wire this prompt as a synchronous API call within your query preprocessing service. The input payload should include the original user query, a list of correction candidates, and any available context such as session history, user role, or domain glossary. The output schema must be validated before retrieval execution. Use a JSON schema validator to confirm each candidate object contains rank, corrected_query, confidence_score (a float between 0.0 and 1.0), and rationale. Reject any response where scores fall outside the expected range or where the top-ranked candidate is missing. Log the full prompt request and response for audit and debugging, but ensure PII redaction occurs before logging if user queries may contain personal data.

Model choice and latency: This prompt benefits from models with strong calibration and reasoning capabilities. For low-latency pipelines, consider a smaller fine-tuned model that has been trained on your correction candidate distributions. For high-stakes domains like clinical or legal retrieval, use a larger model and budget for the additional latency. Implement a timeout and a fallback policy: if the scoring call fails or times out, fall back to a deterministic heuristic such as selecting the candidate with the highest edit-distance similarity to the original query, or executing the top-N candidates in parallel and merging results. Never block the user on a scoring failure.

Calibration and evals: Before production deployment, run the prompt against a held-out ground truth dataset of query-correction pairs. Measure whether the highest-confidence candidate matches the known correct query. Track calibration error by bucketing predictions by confidence score and comparing to actual accuracy in each bucket. A well-calibrated prompt should show that candidates scored at 0.9 are correct roughly 90% of the time. If the prompt is overconfident or underconfident, adjust the temperature, add few-shot examples that demonstrate appropriate uncertainty, or introduce a calibration wrapper that maps raw scores to empirical probabilities using isotonic regression on your eval set.

Human review integration: For high-stakes pipelines, add a gating threshold. If the top candidate's confidence score falls below a configurable threshold (e.g., 0.7), route the original query and all candidates to a human review queue instead of executing retrieval automatically. The review interface should display the original query, the ranked candidates with rationales, and a one-click approval or override action. Log the reviewer's decision to close the feedback loop and improve future scoring. This pattern is essential for regulated domains where incorrect query correction can lead to retrieving irrelevant or harmful documents.