Retrieval Failure Mode Detection Prompt

This prompt is for RAG pipeline operators and reliability engineers who need to move from 'retrieval looks bad' to a specific, actionable failure mode. Instead of manually inspecting vector scores or reading through irrelevant passages, this prompt classifies the failure into a specific mode such as query-document mismatch, missing index coverage, stale data, or ambiguous query intent. Use it when a user query produces low-quality retrieval results and you need a structured diagnosis to decide whether to rewrite the query, re-index documents, update the knowledge base, or route to a fallback. The prompt belongs in automated retrieval debugging pipelines, observability dashboards, and pre-generation quality gates.

The prompt expects a user query and the top-k retrieved passages as input. It outputs a structured classification with a primary failure mode, a confidence score, and a diagnostic evidence snippet that explains why the mode was selected. This is not a replacement for retrieval evaluation benchmarks or human relevance judgment; it is a diagnostic classifier that speeds up root-cause analysis. Wire it into your pipeline after retrieval and before answer generation, or use it offline to triage clusters of bad retrieval traces. For high-stakes applications where a misdiagnosis could route users to the wrong fallback, always pair the classifier output with a human review step or an LLM-as-judge confirmation pass.

Do not use this prompt when you need a fine-grained relevance score for every passage—that is a ranking task, not a failure-mode classification. Do not use it when the retrieval set is empty; the prompt requires at least one retrieved passage to compare against the query. If your system already has structured error codes from the retriever itself, use those signals as additional context fields in the prompt to improve classification accuracy. The next section provides the copy-ready template you can adapt with your own failure taxonomy and output schema.

This prompt template is designed to be dropped directly into your retrieval debugging workflow. It forces the model to act as a diagnostic classifier, moving beyond vague 'bad retrieval' assessments to identify the specific failure mode—such as query-document mismatch, stale data, or ambiguous intent. Replace each square-bracket placeholder with data from your production trace before sending it to the model. The output is a structured classification that can be logged, aggregated, and used to trigger automated remediation in your RAG pipeline.

textCopy
You are a retrieval pipeline diagnostician. Your task is to analyze a failed retrieval attempt and classify the failure into a specific mode. Do not provide a generic assessment. You must identify the most likely root cause from the categories below and support your classification with direct evidence from the provided data.

## FAILURE MODE CATEGORIES
- **query_document_mismatch**: The query terms do not semantically or lexically match the content in the retrieved documents.
- **missing_index_coverage**: The information required to answer the query is not present in the search index at all.
- **stale_data**: The retrieved documents contain information that is factually outdated or has been superseded.
- **ambiguous_query_intent**: The user's query is too vague, has multiple interpretations, or lacks the specificity needed for precise retrieval.
- **retrieval_configuration_error**: The failure is likely due to search parameters (e.g., top_k too low, similarity threshold too strict, incorrect embedding model).
- **document_chunking_artifact**: The relevant information is split across chunks, and no single chunk contains the complete context needed.

## INPUT DATA
**User Query:** [USER_QUERY]
**Retrieved Documents (top_k):** [RETRIEVED_DOCUMENTS]
**Search Configuration:** [SEARCH_CONFIG]
**Expected Answer (if known):** [EXPECTED_ANSWER]

## OUTPUT SCHEMA
You must respond with a valid JSON object conforming to this schema:
{
  "failure_mode": "string (one of the categories above)",
  "confidence_score": 0.0-1.0,
  "diagnostic_evidence": {
    "query_analysis": "string (brief analysis of the query's clarity and specificity)",
    "document_analysis": "string (summary of what the retrieved documents actually contain)",
    "gap_explanation": "string (explanation of why the retrieved content fails to meet the query's need)"
  },
  "recommended_action": "string (a specific, actionable step to remediate this failure, e.g., 'rewrite query to include date filter', 're-ingest source X', 'increase top_k to 20')"
}

## CONSTRAINTS
- If you cannot determine the failure mode with certainty, set the confidence score low and explain what additional information you would need.
- Base your analysis strictly on the provided input data. Do not invent facts about the user or the index.
- The recommended_action must be specific enough for an engineer to implement without further clarification.

To adapt this prompt for your environment, start by populating the RETRIEVED_DOCUMENTS placeholder with the actual text of the top-k results, not just metadata or IDs. The model needs the content to diagnose mismatches. If your pipeline uses a hybrid search, include the SEARCH_CONFIG as a structured object detailing the vector, keyword, and fusion parameters. For high-stakes applications, wire the output confidence_score into a routing rule: scores below 0.7 should trigger a human review of the diagnostic before any automated remediation is applied. You can also extend the FAILURE MODE CATEGORIES list with modes specific to your domain, such as access_control_filter_error or multilingual_mismatch.

This prompt is designed to be called as a post-retrieval diagnostic step, not as a user-facing response generator. Wire it into your RAG pipeline immediately after the retrieval step and before answer generation. The harness should take the original user query, the list of retrieved document chunks with their metadata, and any available retrieval scores as inputs. The prompt's structured failure mode classification should be logged, metered, and used to trigger automated remediation workflows such as re-writing the query, expanding the index, or flagging stale documents for re-indexing.

Implement a strict JSON schema validator on the model's output before the classification enters your observability pipeline. The expected output shape includes a failure_mode enum, a confidence float, and a diagnostic_evidence array of strings. If the model returns malformed JSON, a missing required field, or an unrecognized failure mode, retry once with a repair prompt that includes the raw output and the schema. If the retry also fails, log the raw output and emit an unknown_failure event to your monitoring system. For high-throughput systems, consider using a cheaper, faster model for this classification step (such as a fine-tuned classifier or a smaller LLM) and reserve a more capable model for the answer generation stage.

Integrate the failure mode classification into your existing metrics dashboards. Track the distribution of failure modes over time to detect regressions in retrieval quality after index updates or query pattern shifts. Set alerts on spikes in query_document_mismatch or stale_data modes. For missing_index_coverage failures, automatically append the query to a knowledge gap log that feeds your content acquisition or indexing backlog. For ambiguous_query_intent failures, route the query to a clarification module that asks the user a disambiguating question before re-running retrieval. Always log the full diagnostic payload alongside the trace ID so that on-call engineers can replay and debug specific failures without reproducing the entire session.