Conversation History Summarization Prompt for Token Savings

This prompt is designed for a specific job: converting a long, multi-turn conversation transcript into a dense, structured summary that can replace the raw history in subsequent model calls. The primary user is a chat application developer or AI engineer managing a stateful session where the accumulated turns threaten to exceed the model's context window. Instead of relying on naive truncation—which silently drops the oldest messages and often discards critical early context—this prompt instructs the model to perform a semantic compression. It preserves key facts, user-stated preferences, decisions made, action items, and any questions that remain unresolved. The output is a memory artifact that acts as a drop-in replacement for the raw transcript, dramatically reducing token consumption while maintaining the illusion of a continuous, informed conversation.

The ideal trigger for this workflow is a pre-flight token check. Implement a counter in your application layer that sums the tokens of your system prompt, the raw conversation history, any retrieved context, and the expected output tokens. When this total exceeds a defined safety threshold—typically 80-90% of the model's context limit—you should invoke this summarization prompt before assembling the final request. The summary it produces becomes the new [CONVERSATION_HISTORY] variable in your prompt template. This is not a one-time operation; in very long sessions, you may need to summarize the history, then later summarize the combination of the old summary and new turns. This recursive summarization pattern is a standard technique for unbounded conversational agents, but it requires careful fidelity checks to prevent error propagation.

There are clear situations where this prompt is the wrong tool. Do not use it when the raw conversation easily fits within the context budget; the summarization step itself consumes tokens and introduces a small but non-zero risk of information distortion. Do not use it when the downstream task requires verbatim recall of specific user or assistant phrasing, such as in legal depositions, customer complaints requiring exact wording for a regulatory filing, or debugging sessions where precise error messages matter more than semantic gist. In these cases, you need a different memory strategy, such as a vector database for exact string matching or a longer-context model. Finally, do not use this prompt as a substitute for a proper persistence layer; the summary is a transient memory artifact for the model's context window, not a durable record of the conversation for audit or analytics. Always store the full transcript in your application database separately.

The following prompt template is the core instruction set for compressing a long conversation history into a dense, structured summary. Its primary job is to preserve critical facts, decisions, and open questions while discarding conversational filler, all to save tokens in subsequent LLM calls. The output is designed to be a drop-in replacement for the raw history in a new prompt, making it ideal for long-running chat applications, copilots, and agentic loops where the context window is a scarce resource.

textCopy
System: You are a conversation summarizer. Your task is to compress the provided conversation history into a structured JSON summary. The summary must preserve all key facts, decisions, user preferences, and unresolved questions. Discard pleasantries, repetition, and conversational filler. The output will be used by another AI system to continue the conversation, so it must be self-contained and factually accurate.

User: 
Compress the following conversation history into a JSON summary.

<conversation_history>
[CONVERSATION_HISTORY]
</conversation_history>

<output_schema>
{
  "summary": "A dense, one-paragraph narrative summary of the entire conversation, capturing the user's primary goal and the progress made.",
  "key_facts": ["A list of specific, factual statements extracted from the conversation. E.g., 'User's budget is $500', 'Target launch date is Q3 2024'."],
  "decisions": ["A list of decisions made by the user or agreed upon by both parties. E.g., 'Decided to use PostgreSQL for the database', 'User chose the annual billing plan'."],
  "unresolved_questions": ["A list of explicit questions asked by either party that have not yet been answered."],
  "user_preferences": ["A list of stated user likes, dislikes, or stylistic preferences. E.g., 'Prefers concise answers', 'Dislikes technical jargon'."],
  "action_items": ["A list of tasks that need to be done next, either by the user or the assistant. E.g., 'Assistant to provide a code example', 'User to share API key'."]
}
</output_schema>

<constraints>
- Do not invent any information not present in the original conversation.
- If a section has no items, return an empty list.
- The output must be a single, valid JSON object and nothing else.
</constraints>

To adapt this template, replace the [CONVERSATION_HISTORY] placeholder with your raw conversation data, typically formatted as a sequence of user and assistant turns. The output_schema is a strong suggestion; you should modify the fields to match the specific information your downstream application needs. For instance, a customer support bot might need a "ticket_id" and "issue_category" field, while a coding agent might need "current_branch" and "files_modified". The key is to be explicit about the structure you need for the next step in your pipeline. After generating the summary, always validate the JSON structure before using it to replace the full history, and consider a fidelity check where you ask the model if the summary is consistent with the original text.

The summarization prompt should be called as a side-effect step, not inline with the user's request. In a typical chat application, you maintain a full conversation history in your session store. When the accumulated token count of that history exceeds a configured threshold (e.g., 80% of the model's context limit), you trigger the summarization prompt as a background or pre-turn operation. The prompt receives the raw conversation array as [CONVERSATION_HISTORY] and the current [SUMMARY] if one already exists. The output is a new compressed summary that replaces the previous one, and the raw history can be truncated to only the most recent turns that fit alongside the summary. This keeps the working context window small while preserving long-running session state.

The implementation harness must validate the summary before it replaces the existing state. After the model returns a summary, run a fidelity check: extract all claimed facts, decisions, and unresolved questions from the summary, then verify each against the original conversation turns using a secondary verification prompt or a lightweight NLI model. If the fidelity score drops below a configured threshold (e.g., 95% factual recall), discard the summary and retry with a lower temperature or a more explicit instruction to preserve detail. Log the token counts before and after summarization to track compression ratio. A typical implementation stores the summary in the session object alongside a summary_generated_at timestamp and the compression_ratio metric for observability dashboards.

For retry logic, implement a maximum of two retries with exponential backoff. If both attempts fail fidelity checks, fall back to truncating the conversation history by dropping the oldest turns and log a warning for manual review. Never silently accept a summary that fails validation—a degraded summary can cause the assistant to forget critical user context, leading to repeated questions or incorrect answers. In agent loops, call summarization before planning steps to ensure the planner operates on a compressed but accurate session state. Wire the summarization call into your existing prompt observability pipeline so every summary generation is traceable with the input token count, output token count, fidelity score, and retry count attached to the trace.