Synonym Expansion Failure Mode Analysis Prompt

This prompt is for RAG operators and search engineers who are diagnosing why a synonym expansion pipeline is degrading retrieval quality in production. The job-to-be-done is not generating synonyms, but analyzing a specific failure trace—a user query, the expanded terms that were produced, and the resulting retrieval outcome—to produce a categorized root-cause analysis. The ideal user has access to production logs, understands their retrieval architecture, and needs to move from 'recall dropped' to a concrete remediation ticket. Required context includes the original user query, the full set of expanded terms with any confidence scores or sources, the retrieval results before and after expansion, and the domain or taxonomy the system is expected to operate within.

Do not use this prompt for initial synonym generation, for evaluating expansion quality against a golden dataset, or for building the expansion prompt itself. Those are separate workflows covered by sibling playbooks in this pillar. This prompt is specifically for post-mortem analysis of a failure that has already occurred. It assumes you have a concrete trace to inspect. If you are still designing your expansion strategy or running offline evals, start with the Synonym Expansion Eval Check Prompt Template or the Synonym Expansion with Golden Query Set Evaluation Prompt instead. Using this failure analysis prompt without a real production trace will produce speculative categories that do not reflect actual system behavior.

The prompt requires a structured input payload—do not paste raw logs and expect the model to parse them reliably. Pre-process your trace into the [FAILURE_TRACE] placeholder fields: the original query, the expansion output, the retrieval results with relevance judgments, and any metadata about the retrieval backend and domain. The model will produce a categorized failure analysis with trace annotations and remediation suggestions. This output is designed to be consumed by an engineering team, not an end user, so it includes technical categories like over-expansion, term collision, domain mismatch, and context loss. After running the analysis, validate that each identified failure mode is backed by specific evidence from the trace before opening a remediation ticket.

This prompt template is designed for RAG operators and search engineers who need to systematically analyze why a synonym expansion step degraded retrieval quality in production. It expects a specific failure trace as input and produces a categorized analysis identifying over-expansion, term collision, domain mismatch, and context loss patterns. The output includes a structured trace annotation schema and actionable remediation suggestions. Use this template when you have a concrete example of a query where expansion hurt recall or precision, not for general expansion quality monitoring.

textCopy
You are a search quality analyst specializing in synonym expansion failure diagnosis for RAG systems.

Analyze the provided query expansion trace and identify the root cause of the retrieval degradation. Categorize the failure into one or more of the following patterns:
- **Over-Expansion**: Too many synonyms were added, diluting query intent or exceeding the retrieval engine's effective term budget.
- **Term Collision**: An expanded term matched a different concept in the target corpus, pulling in irrelevant documents.
- **Domain Mismatch**: The generated synonyms belong to a different domain, register, or context than the target knowledge base.
- **Context Loss**: The expansion ignored negation, temporal constraints, entity boundaries, or prior conversation context present in the original query.
- **Entity Corruption**: A recognized entity, proper noun, or product name was inappropriately replaced or expanded.

For each identified failure pattern, provide:
1. The specific expanded term or terms that caused the issue.
2. A brief explanation of the mechanism of failure.
3. A severity rating (CRITICAL, HIGH, MEDIUM, LOW).
4. A concrete remediation suggestion (e.g., add a stop-word, adjust a weight, introduce an entity guard, apply a domain constraint).

Annotate the original trace by marking each expansion term with its failure category and severity. Use the following annotation schema:
- `[OK]` for terms that did not contribute to the failure.
- `[FAIL:<CATEGORY>:<SEVERITY>]` for problematic terms.

Finally, provide a summary assessment with an overall severity and a prioritized list of remediation actions.

# INPUT

**Original Query:** [ORIGINAL_QUERY]

**Expanded Terms (with weights/scores):** [EXPANDED_TERMS]

**Retrieval Impact Evidence:** [RETRIEVAL_IMPACT]

**Domain Context (optional):** [DOMAIN_CONTEXT]

**Conversation History (optional):** [CONVERSATION_HISTORY]

# OUTPUT_SCHEMA

Return a JSON object with the following structure:
{
  "analysis": {
    "primary_failure_pattern": "string",
    "secondary_patterns": ["string"],
    "overall_severity": "CRITICAL|HIGH|MEDIUM|LOW",
    "summary": "string"
  },
  "failures": [
    {
      "term": "string",
      "category": "string",
      "severity": "string",
      "mechanism": "string",
      "remediation": "string"
    }
  ],
  "annotated_trace": [
    {
      "term": "string",
      "original_weight": number,
      "annotation": "OK|[FAIL:<CATEGORY>:<SEVERITY>]",
      "notes": "string"
    }
  ],
  "remediation_plan": [
    {
      "priority": number,
      "action": "string",
      "expected_impact": "string",
      "implementation_effort": "LOW|MEDIUM|HIGH"
    }
  ]
}

# CONSTRAINTS

- Do not invent failures. Only flag terms where the provided evidence supports a causal link to retrieval degradation.
- If the evidence is insufficient to determine a root cause, state that explicitly and suggest what additional data would be needed.
- Preserve all original terms in the annotated trace. Do not drop terms.
- Remediation suggestions must be actionable and specific to the failure mechanism. Avoid generic advice like "improve the prompt."
- If domain context is provided, use it to assess domain mismatch. If absent, note that domain mismatch assessment is limited.

To adapt this template, replace the square-bracket placeholders with your production data. [ORIGINAL_QUERY] should contain the exact user query that triggered the expansion. [EXPANDED_TERMS] should include the full list of generated synonyms with their associated weights or confidence scores. [RETRIEVAL_IMPACT] is the critical evidence field: provide before-and-after retrieval metrics, specific irrelevant documents that were pulled in, or user feedback indicating degradation. The [DOMAIN_CONTEXT] and [CONVERSATION_HISTORY] fields are optional but significantly improve diagnosis accuracy when available. For high-stakes production systems, always pair this analysis with a human review step before applying remediation changes to the expansion pipeline. The output JSON schema is designed to be machine-readable so you can aggregate failure patterns across multiple traces and identify systemic issues in your expansion strategy.

This prompt is not a real-time query rewriter. It is a diagnostic tool for operators and search engineers investigating why a specific synonym expansion degraded retrieval quality. The harness should be triggered by a monitoring alert—such as a recall drop, a spike in null results, or user-reported irrelevance—rather than called on every query. The prompt expects a structured trace object containing the original query, the expanded query, the retrieved document set, and relevance judgments. Without this trace data, the analysis will be speculative and low-value.

Wire the prompt into an incident review workflow or a scheduled batch analysis job. The input trace should be assembled from your retrieval logs, embedding the [ORIGINAL_QUERY], [EXPANDED_QUERY], [EXPANSION_TERMS] with their sources and confidence scores, [RETRIEVED_DOCUMENTS] with their ranks and scores, and [RELEVANCE_JUDGMENTS] (human or LLM-as-judge). Validate the trace object against the expected schema before calling the model. If the trace is incomplete, abort the analysis and log a data quality error rather than generating a misleading report. Use a low-temperature model setting (0.1–0.2) to ensure deterministic, repeatable failure categorization. The output should be parsed as JSON and validated against the [OUTPUT_SCHEMA]—specifically checking that each failure category includes at least one concrete example from the trace and that the remediation suggestions are actionable (e.g., 'add domain constraint to expansion prompt' rather than 'improve expansion').

Log every analysis result alongside the trace ID, model version, and prompt version for auditability. If the analysis flags a critical failure mode like semantic inversion from negation mishandling or entity collision, route the finding to a human review queue before applying any automated remediation. Do not use this prompt's output to automatically modify production expansion prompts or term weights without human approval. The remediation suggestions are advisory. The next step is to correlate failure patterns across multiple traces to identify systemic prompt or retrieval pipeline issues, then use the Synonym Expansion Eval Check Prompt to validate any proposed fix against your golden query set before deployment.