Session-Aware Query Rewrite Prompt with User History

This prompt is for conversational RAG engineers who need to resolve a user's follow-up question into a standalone, retrieval-ready query. The core job is to take the current user utterance, the full conversation history, and any previously resolved entities or clarifications, and produce a single, unambiguous query that can be sent to a vector, keyword, or hybrid search backend. It is designed for multi-turn interactions where anaphora ('it', 'they'), ellipsis ('what about the second one?'), and implicit context from earlier turns would otherwise break retrieval. The ideal user is a developer or AI engineer building a stateful RAG application, such as a customer support copilot, an enterprise document assistant, or a research tool that maintains a conversation thread.

Use this prompt when your system maintains a session state and users ask follow-up questions that depend on prior turns. It is appropriate when you have access to a structured conversation log, including user messages, assistant responses, and any entities or filters that were explicitly resolved in previous steps. The prompt is not suitable for single-shot Q&A systems with no session memory, or for applications where each query must be treated as fully independent. It is also not a replacement for a full dialog manager or a long-term user profile store; it operates strictly within the scope of the current session. If your retrieval system requires metadata filters based on user permissions or roles, pair this prompt with a separate permission-aware filter generation step, not inside this rewrite.

Before implementing, ensure you have a reliable way to detect topic shifts and stale context. The prompt includes instructions for the model to signal when the conversation has moved to a new topic and prior context should be discarded. You should also prepare a set of regression tests that cover pronoun resolution, entity carry-over, clarification incorporation, and explicit topic resets. If the rewritten query will be used to retrieve sensitive or compliance-bound documents, add a human review step or an automated guardrail that validates the rewritten query against the user's access scope before execution. Start by running this prompt against a golden dataset of multi-turn conversations and measure whether the rewritten queries retrieve the correct documents compared to a baseline of single-turn, context-free queries.

This template is the core instruction set for a session-aware query rewriter. It forces the model to treat the current user message as part of an ongoing dialogue, resolving ambiguous pronouns, implicit topics, and follow-up clarifications against the provided conversation history. The goal is to produce a single, standalone, and context-rich query that can be sent to a retrieval system without losing the thread of the conversation. The prompt is designed to be stateless from the model's perspective—all necessary state must be passed in via the placeholders.

codeCopy
SYSTEM:
You are a query rewriter for a conversational search system. Your task is to transform the user's latest message into a standalone, self-contained query suitable for a retrieval engine. You must resolve all anaphora, ellipsis, and implicit references using the provided conversation history and user context. Do not answer the user's question. Output only the rewritten query.

USER CONTEXT:
Role: [USER_ROLE]
Access Scope: [USER_ACCESS_SCOPE]

CONVERSATION HISTORY:
[CONVERSATION_HISTORY]

CURRENT USER MESSAGE:
[USER_MESSAGE]

CONSTRAINTS:
- Resolve all pronouns (e.g., 'it', 'they', 'he') to their specific entities from the history.
- Expand incomplete phrases (e.g., 'What about the second one?') with the full subject.
- If the user corrects or clarifies a previous turn, incorporate the correction and drop the invalidated context.
- If the topic has clearly shifted, ignore irrelevant history and focus only on the new topic.
- Maintain the user's original language and terminology where possible, but expand acronyms on first use in the rewritten query.
- Do not add information not present in the history or current message.
- Respect the user's access scope; do not introduce terms or entities that would require broader permissions.

OUTPUT:
Provide only the rewritten query string, with no preamble or explanation.

To adapt this template, replace the square-bracket placeholders with data from your application's session store. [CONVERSATION_HISTORY] should be a structured log of prior turns, ideally formatted as User: ... Assistant: ... pairs. [USER_ROLE] and [USER_ACCESS_SCOPE] are critical for enterprise deployments where the rewriter must not hallucinate terms from documents the user cannot access. If your system does not use access controls, you can remove those lines and the final constraint. For high-stakes applications, add a [STALE_TOPIC_INDICATOR] flag to force a context reset when the session has been idle beyond a threshold. The next step is to wire this prompt into an application harness that validates the output is a non-empty string and is distinct from the raw user input before sending it to your retrieval pipeline.

The session-aware query rewrite prompt is not a standalone component—it is a preprocessing step that sits between the user's conversational input and your retrieval engine. In a production RAG application, this prompt should be called before every retrieval call in a multi-turn session. The harness must manage session state, feed the prompt with the correct conversation history window, validate the rewritten query, and handle failures gracefully without breaking the user experience. A typical implementation wraps this prompt in a query rewriter service that accepts the current user message, session ID, and user profile, then returns a standalone, permission-aware query string ready for your vector or keyword index.

State Management and Context Assembly: The prompt requires [CONVERSATION_HISTORY], [CURRENT_QUERY], and [USER_PROFILE]. Your harness must assemble these from your session store. For [CONVERSATION_HISTORY], maintain a sliding window of the last N turns (typically 5–10) in a structured format: [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]. Include resolved entities and clarifications from prior turns as metadata. For [USER_PROFILE], inject only the fields relevant to retrieval scoping—role, department, access level, and recent document interactions—not the full user record. Stale context detection is critical: if the session has been idle beyond a configurable TTL (e.g., 30 minutes), flush the history or prompt the user to confirm topic continuity before rewriting. Implement a topic shift detector as a lightweight pre-check: compare the embedding of the current query against the centroid of recent turns; if cosine similarity drops below a threshold (e.g., 0.3), treat the query as a new topic and clear the history window.

Validation and Guardrails: The rewritten query must pass validation before hitting retrieval. Implement a schema validator that checks: (1) the output is a non-empty string, (2) it does not contain unresolved anaphora (e.g., 'it', 'that', 'they') without explicit referents, (3) it does not inject terms that would expand access beyond the user's permission boundary. For permission enforcement, maintain a deny-list of document scopes or metadata filters derived from the user's entitlements. After the rewrite, run a permission boundary check: tokenize the rewritten query and flag any term that maps to a restricted scope not in the user's allow-list. If a violation is detected, either strip the term and re-rewrite or escalate for human review depending on the sensitivity of the data domain. Log every rewrite with the original query, rewritten query, session ID, and any validation failures for auditability.

Retry and Fallback Logic: Model calls can fail, return malformed JSON, or produce rewritten queries that fail validation. Implement a retry loop with exponential backoff (3 attempts maximum). On each retry, include the validation error message in the prompt context so the model can self-correct. If all retries are exhausted, fall back to a degraded rewrite: use the current query as-is after basic normalization (lowercasing, stop-word removal) and append the user's role-based metadata filters directly. This ensures retrieval continues, albeit with reduced context awareness. For high-stakes domains like healthcare or legal, if the rewrite fails validation and retries are exhausted, route the query to a human review queue instead of proceeding with a degraded rewrite. Model choice matters: use a faster, cheaper model (e.g., GPT-4o-mini or Claude Haiku) for the rewrite step, reserving larger models for the final answer generation where quality impact is higher.

Observability and Regression Testing: Every rewrite must be logged with a trace ID that links the original query, rewritten query, session context, validation results, and retrieval performance downstream. This trace data is essential for debugging bad retrievals—when an answer is poor, you need to know whether the rewrite introduced the error. Before deploying any change to the prompt template, run it against a golden dataset of session transcripts with known-good rewrites. Your regression suite should include: (1) follow-up questions with anaphora ('What about the pricing?'), (2) topic shifts mid-session ('Actually, let's talk about security instead'), (3) stale context scenarios where the user returns after a long pause, and (4) adversarial inputs attempting to escape permission boundaries. Automate these tests in your CI pipeline and gate prompt changes on passing the full suite. If your RAG system serves multiple tenants, ensure your test suite includes tenant-isolated sessions to catch cross-tenant context leakage.