Follow-Up Answer Synthesis Prompt with Prior Context

This prompt is designed for RAG assistants operating in multi-turn conversations where a user's current question explicitly depends on the assistant's previous answer. The core job-to-be-done is maintaining cross-turn coherence: the model must connect new evidence to prior claims, resolve anaphora like 'it', 'that', or 'the second one', and produce a response that reads as a logical continuation of the conversation rather than a disconnected factoid. The ideal user is an AI engineer building a copilot, support chatbot, or research assistant where follow-up questions are common and answer consistency across turns is a product requirement. Required context includes the full text of the prior assistant answer, the current user question, and the newly retrieved evidence for the current turn.

Deploy this prompt when the user's question contains explicit back-references to information the assistant just provided. For example, after an assistant describes three pricing tiers, the user asks 'What are the limits on the second one?'—this is a textbook case. The prompt is also appropriate when the new question logically extends the prior answer, such as asking for deeper detail on a specific claim the assistant made. The key signal is dependency: if you cannot answer the current question without access to the prior assistant response, this prompt is the right tool. In your application harness, you should gate this prompt behind a follow-up detection classifier. If the classifier labels the turn as a new topic or standalone question, route to a simpler RAG prompt instead. This prevents the model from fabricating connections to prior context that don't exist.

Do not use this prompt for standalone questions, abrupt topic shifts, or sessions where the prior answer is unavailable, stale, or irrelevant. Forcing the model to reference prior context when none is needed often degrades answer quality—the model may hallucinate a connection or carry forward outdated claims. Similarly, avoid this prompt when the prior answer was an abstention or clarification request; building on an 'I don't know' creates a fragile foundation. If your system detects a topic shift, use a context reset prompt and re-retrieve evidence fresh. For high-stakes domains like healthcare or legal, always pair this prompt with a cross-turn consistency eval that checks whether the new answer contradicts any factual claim from the prior turn. If a contradiction is detected, flag for human review rather than silently overwriting the prior answer.

This prompt template is designed for RAG assistants that must answer follow-up questions coherently across multiple turns. It forces the model to reference the prior answer explicitly, identify what new information is needed, and ground every new claim in the retrieved evidence. The template uses square-bracket placeholders that your application must populate before sending the request. Each placeholder maps to a specific data source in your conversation state, retrieval pipeline, or output schema.

textCopy
You are a precise answer synthesis assistant operating in a multi-turn conversation. Your task is to answer a follow-up question by building on the prior answer while grounding all new claims in the provided evidence.

## PRIOR TURN CONTEXT
Previous question: [PRIOR_QUESTION]
Previous answer: [PRIOR_ANSWER]
Sources cited in prior answer: [PRIOR_CITATIONS]

## CURRENT TURN
Follow-up question: [FOLLOW_UP_QUESTION]
Retrieved evidence for this turn: [RETRIEVED_EVIDENCE]

## INSTRUCTIONS
1. Identify what the follow-up question is asking beyond the prior answer. State this explicitly: "Building on the previous answer about [TOPIC], you're now asking about [NEW_ASPECT]."
2. Check whether the prior answer already contains information relevant to this follow-up. If yes, reference it directly: "As noted in the previous answer, [RELEVANT_PRIOR_POINT]."
3. Ground every new claim in the retrieved evidence. Cite sources inline using the format [Source: DOC_ID].
4. If the retrieved evidence contradicts the prior answer, flag the conflict explicitly: "Note: The new evidence from [DOC_ID] differs from what was stated earlier regarding [SPECIFIC_POINT]. Here's what the current sources indicate."
5. If the follow-up question cannot be answered from the provided evidence, state what is missing and what partial information is available.
6. Do not repeat the full prior answer. Only include prior information when it is directly relevant to the follow-up.

## OUTPUT FORMAT
[OUTPUT_SCHEMA]

## CONSTRAINTS
[CONSTRAINTS]

Adaptation guidance: Replace [PRIOR_QUESTION], [PRIOR_ANSWER], and [PRIOR_CITATIONS] with the actual previous turn data from your conversation store. Populate [FOLLOW_UP_QUESTION] with the current user message and [RETRIEVED_EVIDENCE] with passages from your retrieval pipeline, including document IDs for citation. Define [OUTPUT_SCHEMA] with your expected response structure—this could be a JSON schema, a markdown template, or field descriptions. Use [CONSTRAINTS] to specify domain-specific rules such as citation format requirements, prohibited language, or length limits. For high-stakes domains like healthcare or legal, add an explicit instruction requiring the model to flag low-confidence claims and recommend human review. Before deploying, run this prompt through your eval harness with test cases that include: follow-ups that depend on prior answers, follow-ups that introduce new topics, and follow-ups where new evidence contradicts prior sources. Watch for failures where the model repeats the prior answer verbatim, ignores the prior context entirely, or fails to flag contradictions.

The Follow-Up Answer Synthesis Prompt is not a standalone component; it is a stateful node in a conversational RAG pipeline. The application must maintain a session object that stores the prior turn's question, answer, and the evidence chunks used to produce that answer. Before invoking this prompt, the harness should retrieve new evidence for the current follow-up question, assemble the [PRIOR_QUESTION], [PRIOR_ANSWER], [PRIOR_EVIDENCE], [CURRENT_QUESTION], and [CURRENT_EVIDENCE] variables, and inject them into the template. The model choice matters: use a model with strong instruction-following and long-context handling (e.g., gpt-4o, claude-3.5-sonnet) because the prompt requires the model to simultaneously reason over two evidence sets and a prior answer without confusing which facts belong to which turn.

Validation should happen in two layers. First, a structural validator checks that the output contains the required follow_up_answer and cross_turn_coherence fields, that citations in the answer reference only [CURRENT_EVIDENCE] identifiers, and that the coherence block explicitly flags any contradictions with the prior answer. Second, a semantic evaluator (an LLM judge prompt from the Conversational RAG Evaluation Rubric) scores the output on three axes: grounding fidelity (does the answer stay within current evidence?), cross-turn consistency (does it correctly reference and build on the prior answer without contradiction?), and citation accuracy (are claims properly attributed?). If the semantic score falls below a threshold (e.g., 4/5 on grounding), the harness should trigger a retry with the same inputs and a stronger instruction to avoid fabrication. After two failed retries, escalate to a human review queue rather than surfacing a potentially hallucinated answer to the user.

Logging is critical for debugging cross-turn failures. Capture the full prompt payload, the raw model output, the validator results, and the semantic eval scores for every turn. When a user reports that the assistant contradicted itself, the trace should let you replay the exact prior answer and evidence that were injected into the follow-up prompt. Avoid a common failure mode: injecting the prior answer's full text without its source evidence. If the model sees the prior answer but not the evidence that grounded it, it cannot reliably detect contradictions. Always include [PRIOR_EVIDENCE] alongside [PRIOR_ANSWER]. For high-stakes domains (healthcare, legal, finance), add a human approval step before the follow-up answer is shown to the user whenever the coherence check flags a contradiction or the semantic score is marginal. This prompt is a synthesis step, not a retrieval step—ensure your retrieval pipeline has already fetched fresh evidence for the current question before this prompt runs.