Follow-Up Query Expansion Prompt with History

This prompt is for retrieval engineers and RAG system builders who need to transform a user's follow-up question into a standalone, search-optimized query. The core job-to-be-done is resolving anaphora and implicit context from a multi-turn conversation into a concrete retrieval query without hallucinating constraints that were never stated. The ideal user is an AI engineer integrating this prompt into a retrieval pipeline where a vector database, search index, or hybrid retrieval system expects a clean, decontextualized query string. You need the full conversation history and the current user message as inputs. Do not use this prompt when the user's question is already a complete, self-contained query, or when you are generating the final answer rather than preparing for retrieval.

This prompt is a pre-retrieval step, not a generation step. It should be wired directly into your query rewriting stage, before any call to a vector store or search API. The output is a single expanded query string, not a conversation reply. The primary failure mode is over-expansion, where the model adds assumed constraints like date ranges, filters, or entity names that the user never mentioned. To guard against this, you must validate the output by checking that every term in the expanded query has a direct antecedent in the conversation history or the current question. A secondary failure mode is under-expansion, where the model fails to resolve a pronoun or reference, leaving the query too vague to retrieve relevant documents. Implement a simple validation check: if the output query contains unresolved pronouns or demonstratives like 'it,' 'that,' or 'this,' reject it and retry.

This prompt is not a substitute for a full conversational agent. It does not decide whether retrieval is necessary, nor does it handle clarification requests. Use it as a narrow, testable component in your retrieval pipeline. Before deploying, build a golden dataset of conversation turns and expected expanded queries, and run regression tests to measure query drift and over-expansion rates. If your use case involves high-stakes domains like healthcare or legal research, add a human review step for any query that introduces new constraints not explicitly present in the history.

The following prompt template is designed to be dropped into a retrieval pipeline immediately after a user sends a follow-up message. Its job is to rewrite the user's latest utterance into a self-contained search query that incorporates necessary context from prior conversation turns. The template uses square-bracket placeholders for all dynamic inputs, making it straightforward to populate from your application's session store and retrieval configuration.

textCopy
You are a query rewriter for a retrieval system. Your task is to convert a user's follow-up question into a standalone search query that can be understood without the conversation history.

## Conversation History
[HISTORY]

## Latest User Message
[USER_MESSAGE]

## Instructions
1. Resolve all pronouns, anaphora, and implicit references in the latest message by substituting them with explicit entities or concepts from the conversation history.
2. If the latest message is a clarification, correction, or refinement of a prior question, incorporate that intent into the expanded query.
3. Do not introduce new constraints, entities, or assumptions that are not present in the history or the latest message.
4. If the latest message is a complete topic shift that does not depend on history, return it unchanged.
5. Output only the expanded query text. Do not include explanations, prefixes, or formatting.

## Expanded Query

To adapt this template, replace [HISTORY] with a serialized representation of prior turns—typically the last 3–5 user and assistant message pairs, formatted as User: ... Assistant: .... Replace [USER_MESSAGE] with the raw text of the current user input. For production use, add a [MAX_QUERY_LENGTH] constraint if your search backend has token limits, and consider injecting [DOMAIN_GLOSSARY] as a list of canonical entity names to normalize terminology. Always validate the output before passing it to retrieval: if the expanded query is identical to the raw user message when history is non-empty, the rewrite likely failed and should trigger a retry or fallback to the original message.

This prompt is designed to sit between the user's latest message and your retrieval system. It takes the current query, a window of conversation history, and optionally the last retrieved context, then produces a self-contained search query. The harness should treat this as a pre-retrieval transformation step: the model's output is never shown to the user directly. Instead, it feeds into your vector database, keyword index, or hybrid search. This separation means you can iterate on the expansion prompt without changing your answer-generation logic, and you can log expanded queries independently for debugging.

Wire the prompt into your application with a validation layer before the query hits retrieval. At minimum, check that the output is non-empty, is not a verbatim copy of the user's raw input (which signals the model didn't expand), and does not exceed a reasonable token length for your search backend. For stricter control, implement a drift detector: compare the expanded query against the original using embedding similarity. If the cosine similarity drops below a threshold (e.g., 0.4), the expansion may have introduced unrelated concepts. In that case, fall back to the raw user query or flag the turn for review. Log the original query, the expanded query, the similarity score, and the conversation turn ID for trace analysis.

Model choice matters here. This is a latency-sensitive, low-creativity task. Use a fast, smaller model (e.g., GPT-4o-mini, Claude Haiku, or a fine-tuned open-weight model) rather than a large reasoning model. The prompt is structured for single-turn completion, so avoid models that overthink or produce chain-of-thought unless you explicitly parse out the final query. Set temperature=0 or very low to keep expansions deterministic. If you're using a model that supports structured outputs, constrain the response to a JSON object with a single expanded_query field to avoid parsing ambiguity. Implement a single retry on validation failure with a slightly higher temperature (0.1–0.2) to break out of degenerate outputs, but stop after one retry to avoid latency spikes.

For multi-turn RAG systems, integrate this prompt into your context assembly pipeline just before retrieval. The conversation history you pass should be a compressed summary of prior turns, not raw dialogue, to keep the prompt's token budget predictable. If your system already maintains a session summary for the answer-generation step, reuse that same summary here. Avoid passing the full retrieved context from prior turns into the expansion prompt unless you've confirmed it improves retrieval quality—it often introduces anchoring bias where the model expands toward previously retrieved documents rather than the user's actual intent. Test this with a small golden dataset of follow-up turns before enabling it in production.

Finally, build eval checks into your harness. For each expanded query, record whether the downstream retrieval returned relevant results (measured by answer quality or user feedback). Track the rate of no_results or low-relevance retrievals that follow an expansion. If expansion consistently produces worse retrieval than the raw query, your prompt is over-expanding or hallucinating constraints. Set up a periodic review of a sample of expanded queries—especially those with low similarity scores or high token counts—to catch drift before it degrades the user experience. This prompt is a pipeline component, not a set-and-forget artifact; treat its performance as a metric you monitor alongside retrieval precision and answer accuracy.