Synonym Expansion Eval Check Prompt Template

This prompt is for search quality engineers and RAG developers who need to validate that a synonym expansion system produces correct, relevant terms before shipping to production. Use it when you have a golden set of queries with known-good expansions and you want a structured evaluation report measuring precision, recall, term drift, and regression risk. The prompt assumes you already have an expansion prompt generating candidate terms; its job is to evaluate those outputs, not generate expansions.

The ideal workflow places this eval prompt inside a CI/CD pipeline or a pre-release validation harness. You provide the candidate expansion outputs from your system under test, the golden set of expected expansions, and any domain constraints. The prompt returns a structured JSON report with per-query metrics, aggregate scores, flagged regressions, and a pass/fail recommendation. This lets you gate deployment on measurable quality thresholds rather than manual spot checks.

Do not use this prompt for real-time expansion during user queries—it is an offline evaluation tool. Do not use it when you lack a curated golden set with verified ground-truth expansions, because the eval has no reference point for correctness. Do not use it as a substitute for production observability; it measures pre-release quality, not runtime drift. For high-risk domains such as legal or medical search, always include human review of flagged regressions and edge cases before accepting a pass recommendation.

This template is the core evaluation prompt for synonym expansion systems. It takes a batch of user queries, the expansion terms your system produced, and a golden set of expected expansions, then returns a structured evaluation with per-query metrics, aggregate scores, pass/fail criteria, and regression flags. Use it before deploying expansion changes, when tuning expansion parameters, or as part of a CI pipeline that gates prompt updates.

textCopy
You are a search quality evaluator. Your task is to evaluate synonym expansion outputs against a golden set of expected expansions.

## INPUTS
- User Queries: [QUERIES]
- Generated Expansions: [GENERATED_EXPANSIONS]
- Golden Expected Expansions: [GOLDEN_EXPANSIONS]
- Domain Context: [DOMAIN_CONTEXT]
- Previous Evaluation Baseline (optional): [BASELINE_EVAL]
- Evaluation Thresholds: [THRESHOLDS]

## OUTPUT SCHEMA
Return a JSON object with this exact structure:
{
  "evaluation_id": "string",
  "timestamp": "ISO8601",
  "summary": {
    "total_queries": number,
    "passed": number,
    "failed": number,
    "aggregate_precision": number,
    "aggregate_recall": number,
    "aggregate_f1": number,
    "mean_term_drift_score": number,
    "regression_detected": boolean
  },
  "per_query_results": [
    {
      "query_id": "string",
      "original_query": "string",
      "generated_terms": ["string"],
      "expected_terms": ["string"],
      "true_positives": ["string"],
      "false_positives": ["string"],
      "false_negatives": ["string"],
      "precision": number,
      "recall": number,
      "f1": number,
      "term_drift_score": number,
      "drift_terms": ["string"],
      "passed": boolean,
      "failure_reasons": ["string"]
    }
  ],
  "regression_analysis": {
    "regression_detected": boolean,
    "regressed_queries": ["string"],
    "improved_queries": ["string"],
    "delta_summary": "string"
  },
  "recommendations": ["string"]
}

## EVALUATION RULES
1. For each query, compare generated terms to expected terms using exact string match after lowercasing and stripping whitespace.
2. True positives: terms present in both generated and expected sets.
3. False positives: terms in generated set but not in expected set.
4. False negatives: terms in expected set but not in generated set.
5. Precision = true_positives / (true_positives + false_positives). If denominator is 0, precision = 0.
6. Recall = true_positives / (true_positives + false_negatives). If denominator is 0, recall = 0.
7. F1 = 2 * (precision * recall) / (precision + recall). If both are 0, F1 = 0.
8. Term drift: flag any generated term that is semantically unrelated to the original query or domain context. Score = 1 - (drift_terms_count / total_generated_terms).
9. A query passes if: precision >= [THRESHOLDS.precision_min], recall >= [THRESHOLDS.recall_min], and term_drift_score >= [THRESHOLDS.drift_min].
10. If [BASELINE_EVAL] is provided, compare per-query metrics and flag regression where F1 drops by more than [THRESHOLDS.regression_delta] or previously-passing queries now fail.

## CONSTRAINTS
- Do not hallucinate metrics. Every number must derive from the input data.
- If expected terms are empty for a query, mark recall as 1.0 and note "no golden terms provided."
- If generated terms are empty for a query, mark precision as 0 and recall as 0.
- Flag queries with zero generated terms as "empty expansion" failures.
- Report domain mismatches where generated terms contradict [DOMAIN_CONTEXT].

Adaptation notes: Replace [QUERIES] with a JSON array of query objects, each containing an id and text. [GENERATED_EXPANSIONS] should be a JSON array mapping query_id to arrays of expanded terms. [GOLDEN_EXPANSIONS] uses the same structure with your curated expected terms. [DOMAIN_CONTEXT] is a short string describing the domain (e.g., "enterprise SaaS support tickets") to help detect semantic drift. [THRESHOLDS] is a JSON object with keys precision_min, recall_min, drift_min, and regression_delta. [BASELINE_EVAL] is optional—omit the entire field and the regression section when running the first evaluation. For CI pipelines, parse the summary.passed boolean and regression_analysis.regression_detected flag to gate deployment. Always validate the output against the schema before trusting the metrics.

The Synonym Expansion Eval Check prompt is designed to operate as a gated evaluation step within a continuous integration pipeline, not as an ad-hoc manual check. Its primary function is to compare the output of a candidate expansion prompt against a curated golden dataset of queries and their known-good expansions. The harness must orchestrate three stages: executing the expansion prompt under test across all golden queries, running this eval prompt to score each output, and aggregating results into a pass/fail report. This structured approach prevents regressions in search recall when prompt templates, model versions, or expansion strategies change.

To implement the harness, store the golden dataset as a version-controlled JSONL file where each line contains a query, context (domain, user intent), and expected_expansions (an array of term objects with term and min_weight). The test runner iterates through each entry, calls the expansion prompt under test, and captures the raw output. That output, along with the expected expansions and any domain constraints, is fed into this eval prompt. The eval prompt returns a structured JSON object containing per-query metrics (precision, recall, term_drift_score) and a global pass boolean. The harness must validate this JSON against a strict schema before aggregation. Configure the harness to log the full prompt input, raw model response, and parsed eval output for every test case to enable post-mortem debugging of failures.

Set explicit thresholds for the CI gate: for example, require mean_recall >= 0.85 and mean_precision >= 0.80 across the golden set, with no single query falling below a recall of 0.60. The harness should also detect regressions by comparing the current run's aggregate scores against a stored baseline from the last approved prompt version. If the term_drift_score exceeds a configured threshold (indicating the model is introducing novel but unvalidated synonyms), flag the run for human review even if precision and recall pass. For high-stakes search applications, add a manual approval step in the pipeline that requires a search quality engineer to sign off on the diff report before the new prompt is promoted to production. Avoid running this eval on a model that is significantly weaker than the generation model, as low-quality eval scores will produce noisy gates and erode trust in the pipeline.