Synonym Expansion with Recall-Oriented Evaluation Prompt

This prompt is for search quality engineers and RAG operators who have already deployed a synonym expansion step and need to measure its impact on retrieval recall. It takes a set of test queries, a document corpus, relevance judgments, and the expansion outputs, then produces a structured before-and-after comparison. Use this when you need evidence that your expansion strategy helps rather than hurts, and when you need per-query diagnostics to identify which queries benefit and which degrade.

Do not use this prompt for generating expansions themselves; use the Synonym Expansion Prompt Template for Keyword Search or the Concept Expansion Prompt for Vector Retrieval for that. This prompt assumes you already have expansion outputs and need evaluation, not generation. The ideal user has access to a labeled relevance dataset (qrels), a fixed document corpus, and the output of a synonym expansion pipeline. Without these three inputs, the prompt cannot produce a meaningful comparison. The evaluation is recall-oriented, meaning it measures how many relevant documents were retrieved that would have been missed without expansion. It does not measure precision, ranking quality, or end-to-end answer accuracy—pair it with a separate precision or NDCG evaluation for a complete picture.

Before running this prompt, ensure your relevance judgments are current and representative of real user needs. Stale or sparse judgments will produce misleading recall numbers. Also verify that your expansion outputs are deterministic enough to evaluate; if your expansion prompt introduces high variance, run multiple trials and report confidence intervals. The prompt includes statistical significance notes, but these are approximations—for production gate decisions, supplement with proper statistical tests. Finally, treat per-query recall deltas as diagnostic signals, not final verdicts. A query that degrades after expansion may indicate a domain mismatch, an over-expansion problem, or a judgment gap rather than a broken expansion strategy. Use the output to prioritize manual review of the worst-affected queries before rolling back or tuning your expansion step.

This prompt is the core of the Synonym Expansion with Recall-Oriented Evaluation playbook. It instructs the model to act as a search quality engineer, taking a user query, a set of relevance judgments, and a document corpus to produce a before-and-after recall comparison. The output is a structured report, not just a list of synonyms, making it directly usable in a CI/CD pipeline for search quality regression testing.

textCopy
You are a search quality engineer evaluating the impact of query expansion on retrieval recall.

Your task is to expand a user query with synonyms and related terms, then measure the change in recall against a provided document corpus and relevance judgments.

## INPUT
- User Query: [USER_QUERY]
- Domain Context: [DOMAIN_CONTEXT]
- Document Corpus (ID, Title, Snippet): [DOCUMENT_CORPUS]
- Relevance Judgments (Query ID, Document ID, Relevance Score): [RELEVANCE_JUDGMENTS]
- Expansion Constraints: [EXPANSION_CONSTRAINTS]

## PROCEDURE
1. Generate a list of synonym and related term expansions for the user query, respecting the provided constraints.
2. For each expansion, assign a confidence score (0.0 to 1.0) and a brief rationale.
3. Simulate retrieval using the original query against the document corpus. Record which documents are retrieved.
4. Simulate retrieval using the expanded query (original terms + expansions) against the document corpus. Record which documents are retrieved.
5. Calculate recall@[K] for both the original and expanded queries using the provided relevance judgments.
6. Compute the per-query recall delta.

## OUTPUT_SCHEMA
Return a single JSON object with this exact structure:
{
  "query_id": "string",
  "original_query": "string",
  "expansions": [
    {
      "term": "string",
      "confidence": number,
      "rationale": "string"
    }
  ],
  "original_recall": {
    "recall_at_k": number,
    "retrieved_relevant_docs": ["doc_id"],
    "total_relevant_docs": number
  },
  "expanded_recall": {
    "recall_at_k": number,
    "retrieved_relevant_docs": ["doc_id"],
    "total_relevant_docs": number
  },
  "recall_delta": number,
  "statistical_note": "string describing if the delta is meaningful given the corpus size",
  "failure_flags": ["string describing any issues like zero recall, over-expansion, or missing judgments"]
}

## CONSTRAINTS
- Do not expand named entities, product codes, or proper nouns unless they have known aliases in the domain context.
- Preserve the original query's negation scope. Do not add synonyms that contradict exclusionary terms.
- If no relevance judgments are provided for a document, treat it as non-relevant.
- If the original query retrieves zero relevant documents, flag this in failure_flags.
- Limit expansions to a maximum of [MAX_EXPANSIONS] terms.

To adapt this template, replace each square-bracket placeholder with your application's data. The [DOCUMENT_CORPUS] should be a serialized list of document objects with at least an ID and a snippet for the model to reason over. The [RELEVANCE_JUDGMENTS] are your ground truth, typically a CSV or JSON structure mapping query-document pairs to relevance scores. For high-stakes search systems, always run this evaluation against a held-out golden query set before deploying a new expansion prompt. The failure_flags field is your primary signal for manual review; automate alerts on any non-empty flag array.

This prompt is designed to be the final evaluation gate in a synonym expansion pipeline. It is not a user-facing prompt but a developer-facing evaluation tool. The harness should be invoked automatically after any change to the upstream expansion prompt, the domain glossary, or the retrieval index configuration. The primary job of this harness is to take a batch of queries, their expanded forms from the candidate system, a golden set of relevance judgments, and a document corpus, and produce a structured report comparing recall metrics before and after expansion. The output is a JSON report that downstream CI systems can parse to pass or fail a release.

To integrate this prompt, construct a test runner that iterates over a curated golden query set. For each query, execute the candidate expansion prompt to generate the expanded query, then run both the original and expanded queries against your retrieval system. Collect the top-K document IDs for each. Pass the full matrix—original query, expanded query, original results, expanded results, and the golden relevance labels for each query—into this evaluation prompt as [INPUT]. The prompt expects a structured [INPUT] object containing query_id, original_query, expanded_query, original_results (list of doc IDs), expanded_results (list of doc IDs), and relevance_judgments (a map of doc ID to relevance score). The [OUTPUT_SCHEMA] should be enforced as a strict JSON object with per-query recall deltas, a pass/fail summary, and statistical significance notes. Use a JSON validator in your harness to retry the call if the output does not conform to the schema, and log schema violations for prompt debugging.

For production CI integration, store the golden query set and relevance judgments in version control alongside the prompt templates. The evaluation harness should be triggered on pull requests that modify any file in the prompts/synonym-expansion/ directory. Set a minimum recall improvement threshold (e.g., 5% relative recall gain with p < 0.05) as a release gate. If the evaluation prompt reports a pass status of false or flags significant regressions, block the merge. Log the full evaluation report as a CI artifact for auditability. Avoid running this evaluation on every commit to unrelated code paths; scope the trigger to prompt and retrieval configuration changes to manage compute costs. For high-stakes deployments, add a manual review step that requires a search quality engineer to sign off on the evaluation report before the prompt change reaches production.