Keyword Retrieval Coverage Gap Detection Prompt

This prompt is for search engineers and RAG developers who need to identify systematic blind spots in their retrieval pipeline. The core job-to-be-done is analyzing a production trace where both keyword-based retrieval (e.g., BM25) and semantic (dense vector) search failed to surface a relevant document that a human reviewer or downstream evaluation later confirmed should have been retrieved. The ideal user is someone with access to production traces, including the original user query, the final ranked list of retrieved documents, and a ground-truth 'missed' document. You should use this prompt when you suspect your keyword index has vocabulary gaps, synonym blind spots, or morphological mismatches that semantic search alone cannot compensate for.

This prompt requires a specific set of inputs to be effective. You must provide the original [USER_QUERY], the combined [RETRIEVED_DOCUMENT_LIST] from both keyword and semantic paths, and the [MISSED_DOCUMENT] that should have been returned. The prompt is designed to isolate the failure to the keyword retrieval component, so it is critical that the missed document was also absent from the semantic search results. If the document was found by semantic search but not by keyword search, you should use the 'Hybrid Search Weight Tuning Analysis Prompt' instead. Do not use this prompt for real-time query rewriting; it is a diagnostic tool for offline trace analysis and index improvement.

The output is a structured coverage gap report, not a simple score. It will identify the specific tokens or phrases in the query that failed to match the missed document, classify the gap type (e.g., synonym gap, stemming failure, entity name mismatch), and recommend concrete index or query expansion rules. Before acting on the recommendations, validate them against a broader set of historical queries to ensure the proposed expansion doesn't introduce noise. For high-stakes search applications where a missed document has regulatory or safety implications, always have a human reviewer confirm the gap classification before modifying production indexes.

The following prompt template is designed to be copied directly into your observability or trace analysis workflow. It accepts a production trace containing the user query, the set of documents retrieved by keyword search, the set retrieved by semantic search, and any relevance judgments or user feedback signals. The prompt instructs the model to identify queries where keyword retrieval failed to surface relevant documents that semantic search also missed, producing a structured coverage gap report with actionable recommendations for index or query expansion.

textCopy
You are a search quality engineer analyzing a production retrieval trace. Your task is to detect keyword retrieval coverage gaps: queries where keyword-based retrieval failed to surface relevant documents, and semantic retrieval also did not recover those documents.

## INPUT
- User Query: [USER_QUERY]
- Keyword Retrieval Results: [KEYWORD_RESULTS]
- Semantic Retrieval Results: [SEMANTIC_RESULTS]
- Relevance Judgments (if available): [RELEVANCE_JUDGMENTS]
- User Feedback Signal (if available): [USER_FEEDBACK]

## OUTPUT SCHEMA
Return a JSON object with the following structure:
{
  "coverage_gap_detected": boolean,
  "gap_severity": "none" | "low" | "medium" | "high" | "critical",
  "missing_document_patterns": [
    {
      "pattern_description": string,
      "example_terms_or_phrases": [string],
      "why_keyword_missed": string,
      "why_semantic_missed": string,
      "estimated_relevance": "low" | "medium" | "high"
    }
  ],
  "affected_query_categories": [string],
  "recommendations": {
    "index_expansion": [string],
    "query_expansion_rules": [string],
    "synonym_additions": [string],
    "hybrid_weight_adjustments": string | null,
    "new_fields_to_index": [string]
  },
  "confidence": number,
  "evidence_summary": string
}

## CONSTRAINTS
- Only flag a coverage gap when BOTH keyword and semantic retrieval failed to return a relevant document that should have been retrieved.
- If relevance judgments are unavailable, base your assessment on query-document topical alignment and standard information retrieval expectations.
- Do not flag gaps where the query is ambiguous and multiple interpretations are reasonable without additional context.
- Confidence must reflect the certainty of the gap assessment given available evidence.
- Recommendations must be specific and implementable, not generic advice.

## EXAMPLES
[EXAMPLES]

## RISK LEVEL
[RISK_LEVEL]

To adapt this template, replace each square-bracket placeholder with data from your trace store. [KEYWORD_RESULTS] and [SEMANTIC_RESULTS] should include document IDs, snippets, and relevance scores. [RELEVANCE_JUDGMENTS] can be human labels, click-through data, or explicit feedback. [EXAMPLES] should contain one or two annotated few-shot examples showing a clear coverage gap and a non-gap case to calibrate the model's threshold. [RISK_LEVEL] should be set to "high" if the output drives automatic index changes without human review, or "low" if the report is advisory only. Always validate the output JSON against the schema before ingesting it into your search pipeline or dashboard. For high-risk workflows, route the generated recommendations to a human reviewer before applying index or query expansion changes.

This prompt is designed to be integrated into a batch trace analysis pipeline, not a real-time user-facing feature. The primary integration point is a scheduled job or an on-demand diagnostic tool that queries your observability platform's API for production traces where both keyword and semantic retrieval were attempted but the overall RAG pipeline produced a low-confidence or factually incomplete answer. The prompt expects a structured trace payload containing the user query, the keyword retrieval results, the semantic retrieval results, and the final generated answer. Before calling the model, your harness must assemble this payload from your trace store, ensuring that sensitive or personally identifiable information is redacted from the query and document snippets before they are sent to the model.

The implementation should follow a validate → prompt → parse → log → act loop. First, validate the assembled trace payload against a strict schema: the keyword_results and semantic_results arrays must contain at least one document each, and the user_query must be a non-empty string. If validation fails, log the malformed trace and skip it. After a successful model call, parse the JSON output and validate that the coverage_gaps array contains properly structured objects with gap_type, missed_document_snippet, and recommendation fields. Any parsing failure should trigger a retry with a stricter output format instruction. Log every analysis result—including the trace ID, the number of gaps found, and the specific recommendations—to your observability platform for trend analysis. For high-risk domains like healthcare or legal search, route traces with a high number of critical gaps to a human review queue before automatically applying index changes.

For model choice, a model with strong JSON mode support and a large context window is required because the prompt must ingest multiple sets of search results. GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro are suitable. Avoid small or fast models that may struggle to maintain the structured output format when comparing long document lists. If you are processing a high volume of traces, implement cost controls by first filtering traces to only those where the RAG answer's faithfulness score fell below a defined threshold, or where the user provided negative explicit feedback. Do not run this prompt on every successful trace. The output of this prompt—a list of specific terms and document patterns that keyword search missed—should be programmatically fed into your search index configuration: add the recommended synonyms, adjust the keyword boosting weights, or create new index fields as suggested. Close the loop by monitoring whether subsequent coverage gap analyses show a reduction in the same gap categories.