Conversation Turn Pruning Prompt for Long-Running Sessions

This prompt is for AI engineers and operators who manage conversational AI products with unbounded session growth. The core job-to-be-done is making a structured pruning decision: given a long conversation history that is approaching or exceeding the model's context window, the prompt must identify which turns to keep verbatim, which to summarize, and which to drop entirely. The ideal user is someone who already has a session state manager and needs a reliable, testable pruning step before the next inference call, not someone looking for a generic chat optimization tip.

Use this prompt when you have a concrete conversation history object, a known token budget, and a downstream task that depends on coherent multi-turn context. It is appropriate for customer support copilots, long-running research assistants, and multi-step agent workflows where losing a key user correction or an unresolved question would degrade the assistant's behavior. Do not use this prompt for single-turn classification, RAG over static documents, or sessions under 10 turns where simple truncation is sufficient. The prompt assumes you can provide the full conversation as structured data with speaker roles, timestamps, and turn-level metadata.

Before adopting this prompt, confirm that your application layer can enforce the pruning decision. The prompt produces a plan; your system must execute it by rewriting the session context before the next model call. If you cannot programmatically drop, summarize, or reorder turns based on the output, this prompt adds latency without value. Start by running it against 20 real long-running sessions and manually reviewing whether the pruning decisions preserve task-critical information. If the prompt consistently drops turns that contain unresolved user intent or corrections, adjust the [CRITICALITY_RULES] placeholder to match your domain's definition of what must survive.

This prompt template is designed to be copied directly into your prompt management system or codebase. It instructs the model to act as a conversation archivist, evaluating each turn in a long-running session and assigning a pruning action: keep, summarize, or drop. The template uses square-bracket placeholders for all dynamic inputs, making it safe for string formatting in Python, JavaScript, or any templating engine. Before using it, ensure you have a reliable method for counting tokens in the conversation history and a defined budget for the remaining context window.

textCopy
You are a conversation archivist responsible for managing context window budget in a long-running AI assistant session. Your task is to evaluate each turn in the provided conversation history and assign a pruning decision to reclaim tokens while preserving session coherence.

## Pruning Rules
- KEEP: Turns that contain critical facts, user preferences, unresolved questions, active tasks, or decisions that affect future responses. These turns must remain verbatim.
- SUMMARIZE: Turns that contain useful context but can be compressed into a single sentence without losing essential information. Provide the one-sentence summary.
- DROP: Turns that are greetings, acknowledgments, small talk, resolved sub-tasks, or redundant information already captured elsewhere.

## Constraints
- [CONSTRAINTS]
- The total token count of KEPT and SUMMARIZED turns must not exceed [MAX_OUTPUT_TOKENS] tokens.
- Preserve all explicit user corrections and preference statements as KEEP.
- If a turn contains a multi-step task that is still in progress, KEEP the entire turn.
- Do not drop any turn that introduces a new entity, person, date, or constraint unless it is later explicitly corrected.

## Conversation History
[CONVERSATION_HISTORY]

## Current Context Budget
- Total tokens in conversation history: [TOTAL_INPUT_TOKENS]
- Available token budget for pruned history: [MAX_OUTPUT_TOKENS]

## Output Schema
Return a valid JSON object with the following structure:
{
  "pruned_turns": [
    {
      "turn_index": integer,
      "action": "KEEP" | "SUMMARIZE" | "DROP",
      "content": "Original content if KEEP, one-sentence summary if SUMMARIZE, null if DROP",
      "token_count": integer,
      "rationale": "One-sentence explanation for the decision"
    }
  ],
  "coherence_checks": {
    "unresolved_questions_preserved": boolean,
    "user_preferences_preserved": boolean,
    "active_tasks_preserved": boolean,
    "entity_continuity_maintained": boolean
  },
  "token_summary": {
    "total_tokens_before": integer,
    "total_tokens_after": integer,
    "tokens_reclaimed": integer,
    "budget_compliant": boolean
  }
}

## Output Requirements
- Return ONLY the JSON object. No markdown fences, no explanatory text.
- Every turn in the input must appear exactly once in the pruned_turns array, in the same order.
- If budget compliance is impossible without dropping critical information, set budget_compliant to false and include a note in the final turn's rationale.

To adapt this template, replace each square-bracket placeholder with your runtime values. [CONVERSATION_HISTORY] should contain the full, ordered list of turns with speaker labels and timestamps. [CONSTRAINTS] is where you inject domain-specific rules, such as regulatory retention requirements or mandatory fields that must survive pruning. [MAX_OUTPUT_TOKENS] should be calculated by subtracting your system prompt, tool schemas, and reserved output tokens from the model's context window. If you are using a model with a 128k context window and your system prompt consumes 2k tokens, set [MAX_OUTPUT_TOKENS] to something like 100000 to leave room for the assistant's response. Always validate the assembled prompt's token count before sending it to the model, and log the pruning decisions for observability. For high-stakes sessions, route outputs where budget_compliant is false or coherence_checks fail to a human reviewer before continuing the conversation.

Integrating the conversation turn pruning prompt into a long-running chat application requires a dedicated middleware layer that intercepts the conversation history before it reaches the main reasoning model. The pruning prompt should fire when the token count of the accumulated history exceeds a configured threshold—typically 70–80% of the model's context limit. This threshold must account for the system message, the pruning prompt itself, and the expected output tokens. The pruning decision (keep, summarize, drop) for each turn must be applied deterministically to the history array before the next user request is processed. Do not send the raw pruning output directly to the downstream model; instead, reconstruct a clean history array from the structured pruning result.

Wire the prompt into a dedicated pruning function that accepts a list of conversation turns and returns a pruned list. Each turn should include a unique turn_id, role, content, and a pre-computed token_count. The function constructs the prompt by injecting the serialized history into the [CONVERSATION_HISTORY] placeholder and the available token budget into [MAX_OUTPUT_TOKENS]. The model must return a strict JSON schema: an array of objects with turn_id, action (keep, summarize, drop), and summary (required only when action is summarize). Validate this output before applying it. If the output fails schema validation, is missing required fields, or references non-existent turn_id values, log the failure and fall back to a simpler truncation strategy—such as dropping the oldest turns—rather than sending corrupted history downstream. For high-stakes applications, implement a coherence check: after applying the pruning, run a secondary prompt that asks the model whether any critical information was lost, and escalate to a human reviewer if the confidence score is below a defined threshold.

Model choice matters here. Smaller, faster models (e.g., Claude 3 Haiku, GPT-4o-mini) are well-suited for this classification-style task and keep pruning latency low. Reserve larger models only when the conversation contains complex, nuanced exchanges where summarization quality is critical. Implement retry logic with exponential backoff for transient API failures, but cap retries at three attempts to avoid compounding latency. Log every pruning operation with the original token count, the pruned token count, the model used, the latency, and the full pruning decision array. This data feeds into cost monitoring dashboards and helps tune the pruning threshold over time. Finally, never prune the current user turn or the system message—those are non-negotiable context that must always be preserved.