Conversation History Integration Prompt for RAG

This prompt is designed for RAG system builders who need to integrate multi-turn conversation history with newly retrieved evidence before generating an answer. It solves the core context assembly problem: how to balance what the user said before with what the retrieval system just found, without letting stale dialogue override fresh evidence or losing critical conversational context. The ideal user is an AI engineer or product developer building a copilot, chat assistant, or support agent where answers must remain coherent across a session. You should have a RAG pipeline that has already retrieved passages for the current turn, and you need to produce a single, coherent context block that the answer-generation model can reason over.

Use this prompt when your system must handle follow-up questions that depend on prior answers, resolve anaphora like 'tell me more about the second point,' or maintain factual consistency across turns. It is particularly valuable when the retrieval system returns different evidence for each turn, and you need to merge that new evidence with the established conversational context. For example, a user might ask 'What's the refund policy?' and then follow up with 'How long does that take?' The second question requires the system to understand that 'that' refers to the refund process, and to integrate the newly retrieved processing-time evidence with the previously stated policy context. The prompt template includes placeholders for [CONVERSATION_HISTORY], [CURRENT_QUERY], [RETRIEVED_EVIDENCE], and [OUTPUT_SCHEMA], allowing you to control exactly how history and evidence are weighted.

Do not use this prompt as a standalone answer generator. It is a context assembly step that feeds into a downstream synthesis prompt. If your application only handles single-turn questions, or if you are building a stateless Q&A system where each query is independent, this prompt adds unnecessary complexity. Similarly, if your retrieval pipeline already handles conversation context during the retrieval step itself—for example, by using a query rewriter that incorporates history—you may not need a separate assembly step. For high-stakes domains like healthcare or legal applications, always pair this prompt with human review checkpoints and evidence grounding requirements. The next section provides the copy-ready prompt template you can adapt for your own system.

This prompt instructs the model to act as a context assembly engine. It receives the full conversation history and a fresh set of retrieved passages, then produces a single, deduplicated, and relevance-weighted context block. The output is not the final answer but a structured payload that a separate synthesis prompt can consume. This separation of concerns makes it easier to test, version, and debug each stage independently.

textCopy
You are a context assembly engine for a multi-turn RAG system. Your job is to merge the provided conversation history with new retrieval evidence and produce a structured context block that a downstream synthesis prompt will use to generate the final answer.

## INPUTS

### CONVERSATION HISTORY
[HISTORY]

### RETRIEVED EVIDENCE
[EVIDENCE]

### CURRENT USER QUESTION
[QUESTION]

## INSTRUCTIONS

1. Identify which parts of the conversation history are still relevant to the current question. Ignore resolved topics, abandoned threads, and stale context.
2. From the retrieved evidence, select passages that are directly relevant to the current question and any unresolved context from the history.
3. Deduplicate evidence: if a passage repeats information already established in the conversation history, prefer the history version unless the new passage adds material new facts.
4. Flag any contradictions between the conversation history and the new evidence. Do not resolve them—just mark them.
5. Assign a relevance weight (HIGH, MEDIUM, LOW) to each selected passage based on how directly it addresses the current question and any active context from prior turns.
6. If the retrieved evidence is insufficient to address the question given the conversation history, mark the context block as INCOMPLETE and list what is missing.

## OUTPUT SCHEMA

Return a JSON object with this exact structure:
{
  "active_history_summary": "string summarizing only the still-relevant parts of the conversation",
  "selected_evidence": [
    {
      "passage_id": "string",
      "content": "string",
      "relevance_weight": "HIGH|MEDIUM|LOW",
      "relevance_rationale": "one sentence explaining why this passage was selected"
    }
  ],
  "contradictions": [
    {
      "history_claim": "string",
      "evidence_claim": "string",
      "passage_id": "string"
    }
  ],
  "completeness": "COMPLETE|INCOMPLETE",
  "missing_information": ["string describing what is missing, empty if COMPLETE"],
  "stale_context_discarded": ["string describing discarded history items"]
}

## CONSTRAINTS

- Do not generate the final answer. Only produce the context assembly.
- Do not fabricate evidence. Only use passages from the RETRIEVED EVIDENCE input.
- If the conversation history contains user corrections, treat the corrected version as ground truth.
- Limit selected evidence to a maximum of [MAX_PASSAGES] passages.
- If no evidence is relevant, return an empty selected_evidence array and set completeness to INCOMPLETE.

Adapt this template by adjusting the [MAX_PASSAGES] placeholder to match your context window budget. If your synthesis prompt expects a different schema, modify the OUTPUT SCHEMA block accordingly—but keep the JSON structure strict to enable automated validation. For high-stakes domains, add a [RISK_LEVEL] placeholder that gates whether contradictions trigger an automatic escalation or just a flag. Always validate the output against the schema before passing it to synthesis; a malformed context assembly will cascade into downstream failures.

The conversation history integration prompt is not a standalone component; it is a context assembly step that sits between your dialogue manager and your answer generation prompt. In a production RAG pipeline, this prompt receives the raw conversation history and the newly retrieved evidence, then produces a structured context block that the downstream answer prompt consumes. The harness must enforce a strict contract: the prompt's output must be a valid JSON object with prior_context_summary, resolved_topics, unresolved_topics, relevant_history_turns, and assembled_context fields. Any deviation from this schema breaks the downstream answer prompt, so the harness must validate the output shape before passing it forward.

Wire this prompt into your application as a pre-processing step in the RAG chain. After retrieval returns passages for the current user query, call the conversation history integration prompt with the raw conversation turns and the retrieved chunks. Validate the response against the expected JSON schema using a library like Pydantic or Zod. If validation fails, retry once with a stricter instruction appended to the original prompt: 'You must return only valid JSON matching the specified schema. Do not include explanatory text outside the JSON object.' If the second attempt fails, log the raw output, fall back to a minimal context assembly that includes only the current query and retrieved passages, and alert the on-call channel. For model choice, use a fast, instruction-following model like GPT-4o-mini or Claude Haiku for this assembly step—it is a structuring task, not a reasoning task, and latency matters because this step adds overhead before the main answer generation.

Logging is critical for debugging multi-turn failures. Capture the input token count, output token count, validation status, retry count, and the final assembled context for every call. Include a session_id and turn_index in your log metadata so you can trace context degradation across a conversation. Implement a staleness check: if the assembled context references a prior turn that is more than [MAX_TURN_AGE] turns old or older than [MAX_TIME_WINDOW] minutes, flag it for review. For high-stakes domains, add a human review gate when the prompt's confidence in history relevance drops below a configurable threshold—expose this as a relevance_score field in the output schema and route low-confidence assemblies to a review queue. Do not let this prompt silently inject stale or irrelevant history into the answer generation step; the harness is your guardrail.