Sliding Window Summary Prompt for Chat History

Use the Sliding Window Summary Prompt when you are building a conversational AI application that must maintain coherent state across long, multi-turn sessions that exceed the model's context window. The primary job-to-be-done is preserving critical conversational state—decisions, user preferences, unresolved action items, and key facts—while safely evicting older, less relevant turns. This prompt is designed for AI engineers and application developers who are implementing the memory layer of a chat copilot, support agent, or long-running assistant, not for one-shot Q&A or short sessions that fit comfortably within the context limit.

This prompt is appropriate when the cost of losing conversational context is high. Concrete scenarios include a customer support session spanning dozens of turns where an agent must remember a reported serial number and troubleshooting steps already taken, a coding assistant that needs to recall architectural decisions made earlier in a session, or a research assistant synthesizing findings over a multi-hour collaboration. The prompt template requires you to supply the current summary, the oldest turns to be evicted, and a schema for what must be preserved. You should wire this into your application's state management layer so the summary is updated and persisted after each sliding window operation, not regenerated from scratch on every turn.

Do not use this prompt when the conversation is short enough to fit entirely within the context window, as the summarization step adds latency and a non-zero risk of fact distortion. Avoid it in high-stakes domains—such as clinical handoffs or legal proceedings—where a summary artifact could be mistaken for a complete record; in these cases, a full transcript with human review is required. Also avoid using this prompt as a substitute for a proper long-term memory or personalization system. The sliding window summary is a context management technique for a single session, not a user profile store. Before deploying, build an eval harness that measures context retention accuracy by checking whether evicted facts critical to the task are still recoverable from the summary, and set a threshold below which the system must escalate for human review.

This prompt template implements a sliding window summary strategy for long-running conversational AI sessions. When the conversation history approaches the model's context limit, this prompt instructs the model to compress earlier turns into a structured summary, preserving critical state while freeing token budget for new interactions. The template is designed to be called programmatically by your application harness whenever the estimated token count exceeds a configured threshold.

textCopy
You are maintaining a rolling summary of an ongoing conversation to manage context window limits.

Your task: Read the [CURRENT_SUMMARY] and the [NEW_TURNS] below, then produce an updated summary that preserves all critical information while fitting within [MAX_SUMMARY_TOKENS] tokens.

[CURRENT_SUMMARY]
[EXISTING_SUMMARY_TEXT]

[NEW_TURNS]
[RECENT_CONVERSATION_TURNS]

[RETENTION_RULES]
You must preserve the following categories of information in the updated summary:
- [REQUIRED_CATEGORIES]

[OUTPUT_FORMAT]
Produce the updated summary as a single coherent paragraph with no bullet points or markdown. Begin with "SUMMARY:" followed by the compressed text.

[CONSTRAINTS]
- Do not exceed [MAX_SUMMARY_TOKENS] tokens in the output summary.
- Prioritize recent information over older details when conflicts arise.
- If a category in [RETENTION_RULES] has no content to preserve, omit it silently.
- Do not include meta-commentary about the summarization process.
- Preserve exact values for numbers, dates, identifiers, and code snippets.

To adapt this template for your application, replace each square-bracket placeholder with values from your conversation state manager. The [REQUIRED_CATEGORIES] field should list the specific information types your use case demands—common examples include user preferences, pending action items, decisions made, unresolved questions, and entity references. Wire the output of this prompt back into your session state as the new [CURRENT_SUMMARY] for the next summarization cycle. Test with eval checks that verify category-level retention accuracy before deploying to production.

The Sliding Window Summary Prompt is not a standalone call—it is a state-management step that runs inside a chat loop. When the conversation history approaches the model's context limit, the harness must trigger this prompt before the next user turn. The prompt receives the full conversation history and the last known summary, and it returns a new summary that replaces the evicted turns. The application then constructs the next model request using the new summary plus the most recent turns, keeping the total token count within budget. This means the harness needs a reliable token counter, a configurable threshold (e.g., trigger when 80% of the context window is full), and a place to store the rolling summary between turns.

Wire the prompt into a stateful session object that holds: conversation_history (list of turns), rolling_summary (string or null), summary_metadata (last summary timestamp, turn index covered, token count). Before each user turn, run a preflight check: count tokens in conversation_history plus the system prompt and any RAG context. If the total exceeds the trigger threshold, call the Sliding Window Summary Prompt with the full history and current rolling_summary. Validate the output: check that the returned summary is non-empty, contains fewer tokens than the evicted turns, and preserves key fields (decisions, preferences, unresolved items). If validation fails, retry once with a stronger constraint instruction. If it fails again, log the failure, keep the old summary, and escalate to a human reviewer or fallback model. On success, update rolling_summary, evict summarized turns from conversation_history, and proceed with the user turn.

For production deployment, treat this prompt as a critical state-mutation step. Log every summary generation with the turn range covered, token counts before and after, and a hash of the summary for auditability. Run periodic evals that compare the summary against the original turns to measure information retention—check that decisions, user preferences, and open questions survive the compression. Common failure modes include: the summary growing too large over many cycles (mitigate with a summary-of-summaries step), the model dropping a critical user preference (mitigate with a preference-extraction pre-check), or the summary prompt itself exceeding the context window (mitigate by summarizing in chunks if the history is extremely long). Do not use this prompt for sessions under 20 turns unless token pressure is severe; a simpler truncation strategy is cheaper and faster for short conversations.