Dialogue History Compression Prompt for RAG

This prompt is for operators and engineers managing token budgets in long-running RAG conversations where the context window is filling up and cost or latency is becoming prohibitive. The job-to-be-done is injecting a compressed representation of the dialogue history into the next turn's prompt assembly so the model can maintain conversational coherence without reprocessing every prior exchange. The ideal user is an AI engineer or platform operator who has already instrumented their RAG pipeline to track token consumption per turn and needs a programmatic compression step before the next retrieval-augmented generation call. You should have access to the raw conversation transcript, the previously retrieved evidence chunks, and the system's prior answers before invoking this prompt.

Use this prompt when your conversation has exceeded 10-15 turns, when you observe answer quality degrading due to context truncation, or when your token accounting shows that dialogue history is consuming more than 40% of the available context window. The compression is specifically designed to preserve question-answer pairs, unresolved items, key facts extracted from evidence, and user corrections while dropping redundant turns, filler exchanges, and already-resolved clarifications. Do not use this prompt for general text summarization, for compressing retrieved documents (that's a separate retrieval-side concern), or for single-turn RAG systems where history is irrelevant. It is also not appropriate when the conversation contains legally binding statements, clinical decisions, or compliance-critical exchanges where lossy compression could drop material facts—in those cases, implement a full audit log alongside any compression step.

Before deploying this prompt in production, run a regression test suite that compares answer quality with and without compression across at least 20 multi-turn conversation scenarios. Measure factual consistency against source evidence, citation preservation, and unresolved question carry-forward. Common failure modes include dropping a user correction from 5 turns ago, compressing a clarification question into a statement of fact, or losing the distinction between 'the system couldn't answer this' and 'the system hasn't tried yet.' If your eval shows answer degradation above 5% after compression, reduce the compression ratio or add a human review step before the compressed history is injected into the next turn. Wire this prompt into your prompt assembly pipeline as a pre-processing step that runs when a token budget threshold is breached, not as a user-facing feature.

This prompt template is designed to be dropped into a RAG pipeline where long conversation histories threaten to exceed the model's context window or consume an unsustainable token budget. It instructs the model to act as a lossy compression engine, prioritizing the preservation of semantic content—questions, answers, factual claims, and unresolved threads—over conversational niceties like greetings, acknowledgments, and redundant turns. The output is a structured, machine-readable summary that can be injected into future context windows as a replacement for the raw transcript, maintaining multi-turn coherence without carrying the full weight of the dialogue.

textCopy
Compress the following dialogue history into a dense, structured summary. Preserve every question-answer pair, any unresolved questions, and key factual claims. Drop greetings, acknowledgments, redundant turns, and filler. Output only the compressed history.

[DIALOGUE_HISTORY]

Output the compressed history in the following JSON format:
{
  "summary": "A single paragraph narrative summary of the conversation's core topics and outcomes.",
  "qa_pairs": [
    {
      "turn": <integer>,
      "question": "The user's exact question or request.",
      "answer": "The assistant's core answer, stripped of filler.",
      "citations": ["source_id_1", "source_id_2"]
    }
  ],
  "unresolved_questions": [
    "Question that was asked but not adequately answered."
  ],
  "key_facts": [
    "Factual claim made by the user or assistant that may be relevant later."
  ],
  "user_intent": "The inferred high-level goal of the user in this session."
}

To adapt this template, replace [DIALOGUE_HISTORY] with your raw conversation transcript, typically formatted as a sequence of User: and Assistant: turns. The output schema is a strong starting point but should be modified to match your application's needs. If your RAG system uses a specific citation format, adjust the citations field accordingly. If you do not need intent tracking, remove the user_intent field to save tokens. For high-stakes domains, add a [CONSTRAINTS] placeholder before the output schema to inject rules like 'Do not fabricate citations' or 'Mark any factual claim not grounded in the provided history with [UNVERIFIED].' Always validate the output JSON against your schema before injecting it into the next turn's context window; a malformed summary is worse than no summary.

The dialogue history compression prompt is not a standalone utility—it is a pipeline stage that sits between your conversation store and your next LLM call. In a typical RAG chat application, you maintain a list of conversation turns (user messages, assistant responses, and optionally retrieved evidence or citations). Before each new user message, you check whether the accumulated history plus the new retrieval context would exceed your model's context window or your token budget. If so, you invoke the compression prompt to produce a lossy summary that preserves question-answer pairs, unresolved items, and key facts while dropping redundant or resolved turns. The compressed summary then replaces the raw history in the context assembly step.

Implement this as a pre-call middleware in your LLM request pipeline. The middleware should: (1) calculate the current token count of the full conversation history plus the pending retrieval context and system prompt; (2) compare against a configured max_context_tokens threshold (typically 70-80% of the model's context limit to leave room for the output); (3) if over budget, call the compression model with the raw history and the [COMPRESSION_TARGET_TOKENS] parameter; (4) validate the compressed output before substituting it into the context. Use a separate, cheaper model for compression (such as GPT-4o-mini, Claude Haiku, or a local summarization model) to keep costs predictable. The compression model should be fast and reliable, not necessarily the same model that generates the final answer. Log the compression ratio (original tokens vs. compressed tokens) and the compression timestamp for observability.

Validation and retry logic are critical because a malformed compression can corrupt downstream answer quality. After receiving the compressed summary, run a lightweight validator that checks: (a) the output is non-empty and under the target token count; (b) it contains at least one question-answer pair or unresolved item marker; (c) it does not contain hallucinated facts not present in the original history (a simple overlap check against key entities or a follow-up fact-verification prompt can catch this). If validation fails, retry once with a more explicit instruction (e.g., 'The previous compression was empty. Please include all unresolved questions.'). If the retry also fails, fall back to truncating the raw history by dropping the oldest turns and log a warning. Never silently pass a failed compression into the answer-generation step.

Tool integration is straightforward: the compression prompt is a text-in, text-out operation with no external tool calls. However, if your conversation history includes structured metadata (citation IDs, evidence chunk references, tool call results), you must decide whether to preserve that structure in the compressed summary or flatten it. For most RAG applications, preserving citation IDs and evidence references in the compressed output is essential so the downstream answer generator can still ground claims. Include a [PRESERVE_METADATA] placeholder in the prompt template that specifies which fields to retain (e.g., 'Preserve all citation IDs in square brackets and unresolved question markers'). If your history contains tool-use turns (function calls and results), compress those into a concise 'tool used: [name], result: [summary]' format rather than dropping them entirely.

Human review is not required for every compression call—this is an automated pipeline stage. However, you should build a spot-check dashboard that samples compressed summaries alongside their original histories so operators can verify compression quality during development and after prompt changes. Set up an eval harness that runs the compression prompt against a golden dataset of conversation histories and measures: (1) answer quality preservation (does the downstream RAG answer remain correct when using compressed vs. raw history?); (2) unresolved item retention (are all open questions still present?); (3) compression ratio achieved. Run these evals as part of your prompt CI pipeline before deploying compression prompt changes. If your application operates in a regulated domain, log every compression event with the original history, compressed output, and validation result for audit trails.