Synonym Expansion Harness with Confidence Scoring Prompt

This playbook is for infrastructure teams building production retrieval pipelines who need more than a list of synonyms. The prompt produces expanded terms with per-term confidence scores, source rationale, and a reject threshold so downstream systems can decide whether to use, weight, or discard each suggestion. Use this when your expansion step feeds into a re-ranker, a fusion layer, or an audit log that requires traceability. The primary job-to-be-done is turning a raw user query into a structured, machine-readable expansion payload that a retrieval orchestrator can consume without additional parsing or guesswork.

The ideal user is a search infrastructure engineer, a RAG platform developer, or an ML engineer responsible for query rewriting stages. They operate in an environment where retrieval quality is measured, regressions are caught in CI, and every transformation step must justify itself. This prompt fits when you need to answer questions like: 'Which expanded terms actually improved recall?' or 'Why did this query retrieve those documents?' The confidence scores and source rationale fields make expansion decisions auditable, which matters for regulated domains, A/B test analysis, and debugging retrieval failures. If you are running a one-off search experiment, building a quick prototype, or working in a context where a simple list of synonyms is sufficient, this harness introduces unnecessary complexity. Do not use this prompt when latency budgets are extremely tight and you cannot afford the extra tokens for rationale generation, or when your downstream system cannot consume structured JSON with per-term metadata.

Before implementing, confirm that your retrieval pipeline has a place to consume structured expansion outputs. The prompt assumes a JSON contract with required fields for expanded_terms, confidence_score, source_rationale, and a reject_threshold. If your current pipeline passes raw text strings between stages, you will need to adapt the consumer before this prompt adds value. Start by running the prompt against a golden query set and measuring whether the confidence scores correlate with actual retrieval improvement. If they do not, recalibrate the threshold or adjust the few-shot examples before integrating into production traffic.

This template is the core instruction set for the Synonym Expansion Harness. It is designed to be called as a single-turn task by your application's model gateway. The prompt instructs the model to act as a query expansion engine that not only generates synonyms but also assigns a calibrated confidence score, provides a source rationale for each term, and explicitly rejects low-confidence suggestions. This structured output is critical for downstream decisions, such as whether to use a term in a keyword query, boost it in a hybrid retriever, or discard it to prevent noise.

textCopy
You are a query expansion engine for a production retrieval system. Your task is to expand the user's query with synonyms and conceptually related terms to improve search recall.

INPUT QUERY: [USER_QUERY]
DOMAIN CONTEXT: [DOMAIN_CONTEXT]
OUTPUT SCHEMA: [OUTPUT_SCHEMA]
CONSTRAINTS: [CONSTRAINTS]

For each term in the input query, generate a list of expansion candidates. For each candidate, you must provide:
1.  **term** (string): The expanded word or phrase.
2.  **confidence** (float 0.0-1.0): Your confidence that this term is a relevant synonym or conceptual match in the given domain context. A score below [REJECT_THRESHOLD] means the term should not be used.
3.  **rationale** (string): A brief explanation of the semantic relationship (e.g., 'direct synonym', 'hypernym', 'domain-specific jargon', 'conceptual alternative').

Do not expand named entities, proper nouns, or negation terms. Preserve the original query's intent. If no high-confidence expansion exists for a term, return an empty list for that term.

Return the output strictly as a JSON object matching the provided schema.

To adapt this template, replace the square-bracket placeholders at runtime. [USER_QUERY] is the raw input string. [DOMAIN_CONTEXT] should be a concise string describing the knowledge base's domain (e.g., 'enterprise cloud infrastructure', 'medical research papers on oncology'). [OUTPUT_SCHEMA] must be a valid JSON Schema definition that your application's parser expects; this forces the model into a strict contract. [CONSTRAINTS] is where you inject dynamic rules, such as a maximum token budget per query or a list of protected terms that must not be expanded. The [REJECT_THRESHOLD] is a critical control; set it based on your offline precision-recall evals (a typical starting point is 0.6). After pasting, validate that your orchestration layer can handle a JSON parsing failure by implementing a retry with a simplified schema or a fallback to the original query.

This prompt is designed as a pre-retrieval expansion service that sits between the user's raw query and your search backend. The harness expects a JSON input payload containing the user query, optional domain context, and a confidence threshold. The model returns a structured JSON output with expanded terms, per-term confidence scores, source rationale, and a rejection flag for low-confidence suggestions. The primary integration point is a server-side function that calls the LLM, validates the output against the contract schema, and merges surviving terms into the retrieval query before execution.

Integration flow: 1) Receive the user query from your application. 2) Assemble the prompt with [QUERY], [DOMAIN_CONTEXT], and [CONFIDENCE_THRESHOLD]. 3) Call the model with response_format set to json_object (or equivalent structured output mode) and a low temperature (0.0–0.2) for deterministic scoring. 4) Parse the response and validate each term object against the expected schema: term (string), confidence (float 0–1), rationale (string), and reject (boolean). 5) Filter out any term where reject is true or confidence is below your operational threshold. 6) Append the surviving terms to the original query using your search engine's query syntax (e.g., OR clauses for keyword indexes, weighted concatenation for vector queries). 7) Log the full expansion payload for observability and later eval.

Validation and safety checks: Implement a schema validator that rejects malformed responses before they reach retrieval. Common failure modes include missing confidence fields, non-boolean reject flags, and terms that duplicate the original query verbatim. Add a term collision detector that flags when an expanded term is identical to an existing query token—these should be dropped. For high-recall use cases, set [CONFIDENCE_THRESHOLD] to 0.6–0.7; for precision-sensitive domains like legal or medical search, raise it to 0.8–0.85 and route low-confidence expansions to a human review queue. If the model returns zero terms above threshold, fall back to the original query without expansion and log the event for threshold tuning.

Retry and model selection: This prompt works well with models that support structured JSON output (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro). If the response fails schema validation, retry once with the same prompt and temperature increased to 0.3 to break deterministic failure patterns. If the second attempt also fails, log the raw response, fall back to the original query, and alert the search quality team. For latency-sensitive pipelines, consider caching expansion results for identical [QUERY] + [DOMAIN_CONTEXT] pairs with a TTL of 1–4 hours, as synonym relationships rarely change within a session. Do not cache expansions across different [CONFIDENCE_THRESHOLD] values—the threshold directly affects which terms survive filtering.

Observability and eval integration: Log every expansion call with the original query, the full model response, the post-filter term list, and the retrieval latency delta. Use the companion Synonym Expansion Eval Check Prompt from this pillar to run periodic regression tests against a golden query set. Track per-call metrics: number of terms generated, number surviving threshold, rejection rate, and downstream recall impact. When the rejection rate exceeds 40% or the average confidence drops below 0.5, investigate whether the domain context needs updating or the model is drifting. Wire these metrics into your existing prompt observability dashboard alongside token usage and latency.