Context Window Eviction Strategy Prompt Template

This prompt template is designed for AI architects and platform engineers building long-running agent workflows where the context window is a hard constraint. The core job-to-be-done is to programmatically decide which information to discard when the token limit is reached, ensuring the agent can continue its task without losing critical state. The ideal user is someone integrating this into an agent harness or orchestration framework, not an end-user chatting with a model. You need a structured history of the agent's actions, observations, and a clearly defined goal to use this effectively.

Do not use this prompt for simple chat history summarization or one-off document truncation. It is specifically for multi-step agent tasks where the cost of evicting the wrong information is a derailed plan or a repeated loop. The prompt requires a pre-computed context inventory—a list of messages, tool outputs, or thoughts—each tagged with metadata like recency, type, and dependencies. Without this structured input, the model cannot make a reliable eviction decision. This is a decision prompt, not a summarization prompt; its output is a list of identifiers to remove, which your application code must then execute.

Before implementing this, ensure your agent harness can capture and serialize the full conversation state into the required [CONTEXT_INVENTORY] format. The prompt is a component in a larger control loop: your system detects an impending overflow, invokes this prompt to get an eviction plan, prunes the context, and then re-injects the remaining history into the next model call. The primary risk is evicting a dependency that a future step requires, so your evaluation must test for task continuity after eviction, not just the validity of the eviction plan itself.

This prompt template is designed to be injected into a long-running agent workflow when the context window approaches its limit. Instead of naive truncation—which often discards critical instructions, recent observations, or unresolved dependencies—this prompt instructs the model to analyze the full context, score each element, and produce a compressed continuation context that preserves what matters most. The output is not a final answer but a restructured context payload that can be loaded into a fresh window so the agent can continue without losing state.

textCopy
You are managing a context window that has reached its capacity limit. Your job is to produce a compressed continuation context that fits within [MAX_OUTPUT_TOKENS] tokens while preserving the information most critical for completing the task below.

## TASK
[TASK_DESCRIPTION]

## FULL CONTEXT (TOO LARGE TO CONTINUE)
[FULL_CONTEXT]

## EVICTION RULES
1. Score every context element (messages, tool outputs, observations, instructions) on three dimensions from 1 (low) to 5 (high):
   - RECENCY: How recently was this element added?
   - RELEVANCE: How directly does this element relate to the current task?
   - DEPENDENCY: Do other elements or pending actions depend on this element?
2. Compute a composite score: (RECENCY * [RECENCY_WEIGHT]) + (RELEVANCE * [RELEVANCE_WEIGHT]) + (DEPENDENCY * [DEPENDENCY_WEIGHT]).
3. Preserve ALL elements with a composite score above [PRESERVATION_THRESHOLD].
4. For elements below the threshold, create a one-line summary only if they contain information that might be needed later.
5. Never evict: the original task description, active tool schemas, pending action items, unresolved errors, or user identity and permissions.

## OUTPUT FORMAT
Return a JSON object with this schema:
{
  "preserved_elements": [
    {
      "element_id": "string",
      "content": "string (verbatim preserved content)",
      "composite_score": number,
      "preservation_reason": "string"
    }
  ],
  "summarized_elements": [
    {
      "element_id": "string",
      "summary": "string (one-line summary)",
      "original_score": number
    }
  ],
  "evicted_elements": [
    {
      "element_id": "string",
      "eviction_reason": "string"
    }
  ],
  "continuation_context": "string (assembled context ready for the next window, including preserved content and summaries)",
  "state_checkpoint": {
    "pending_actions": ["string"],
    "unresolved_dependencies": ["string"],
    "critical_state_variables": {"key": "value"}
  }
}

## CONSTRAINTS
- The continuation_context field must be under [MAX_OUTPUT_TOKENS] tokens.
- Preserve exact quotes, numbers, dates, and identifiers without modification.
- If two elements conflict, preserve both and flag the conflict in state_checkpoint.
- If the task cannot be safely compressed below the token limit, set a flag "requires_escalation": true and explain why.

Adaptation guidance: Replace [TASK_DESCRIPTION] with the original task the agent is pursuing. [FULL_CONTEXT] should contain the entire context window contents, typically assembled from message history, tool outputs, and system instructions. Tune [RECENCY_WEIGHT], [RELEVANCE_WEIGHT], and [DEPENDENCY_WEIGHT] based on your domain—for real-time operational agents, recency often dominates; for research agents, relevance and dependency matter more. Set [PRESERVATION_THRESHOLD] between 8 and 12 depending on how aggressively you need to compress. [MAX_OUTPUT_TOKENS] should be your target context window size minus space reserved for the next model response. Before deploying, run this prompt against at least 20 recorded overflow scenarios and verify that the continuation context preserves all elements needed to complete the original task without restarting. For high-risk domains such as healthcare or finance, require human review of the eviction decisions before the agent continues with the compressed context.

The Context Window Eviction Strategy prompt is not a one-shot call; it is a decision node inside a long-running agent harness. When the context window approaches a configured high-water mark (e.g., 80% of the model's maximum context tokens), the harness must pause the primary task, collect the current context inventory, and invoke this prompt to produce a ranked eviction plan. The harness then executes the plan by removing the lowest-scoring items before resuming the primary task. This section covers the wiring, validation, and failure modes you need to handle before deploying this prompt into a production agent loop.

Harness integration steps: 1) Token monitoring: Use the model's token-counting API or a local tokenizer to track total context tokens after each turn. Define a HIGH_WATERMARK threshold (e.g., 0.8 * model_context_limit). 2) Context inventory construction: When the watermark is breached, build a structured inventory of all context items—system prompt segments, tool results, prior turns, retrieved documents, memory entries—each with a unique item_id, content_summary (first 100 chars), age_in_turns, and source_type. 3) Prompt invocation: Populate the [CONTEXT_INVENTORY] placeholder with this inventory and [TASK_DESCRIPTION] with the current agent goal. Call the model with response_format set to JSON and a strict schema requiring { "eviction_plan": [{"item_id": "...", "score": 0.0-1.0, "reasoning": "..."}] }. 4) Validation gate: Before evicting, validate that the returned item_id values exist in the inventory, scores are within range, and the plan does not evict items tagged as [IMMUTABLE] (e.g., system-level safety instructions). Reject invalid plans and retry with an error message injected into the prompt. 5) Eviction execution: Remove items with scores below a configurable EVICTION_THRESHOLD (start at 0.3), log the eviction event with item IDs and scores, and resume the primary agent loop.

Model choice and latency considerations: This prompt requires strong instruction-following and structured output reliability. Use GPT-4o or Claude 3.5 Sonnet for production; avoid smaller models that may produce malformed JSON or hallucinate item IDs. Expect 1-3 seconds of latency per eviction decision. For high-frequency agent loops, consider batching eviction decisions or using a cheaper model for the scoring pass and a stronger model only when the plan is ambiguous. Retry logic: If the model returns invalid JSON, missing fields, or references to non-existent item IDs, retry up to 2 times with the validation error appended to the prompt. After 2 failed retries, fall back to a deterministic eviction strategy (e.g., evict oldest items first) and log the fallback for review. Logging and observability: Record every eviction event with item_id, score, reasoning, timestamp, and task_id. This log is essential for debugging task continuity failures—if the agent loses critical context, you can trace which eviction decision caused the loss and tune the scoring prompt or threshold.

Testing and evaluation: Before production, build an eval harness that simulates context overflow with known critical items (e.g., a user's name, a key constraint, a tool result needed for the final answer). Run the eviction prompt and verify that critical items receive high scores and survive eviction. Measure task continuity: after eviction, ask the agent a question that requires the retained context and check for correct answers. A failure here means the scoring prompt is undervaluing certain context types—update the [SCORING_CRITERIA] section of the prompt template to weight those types higher. What to avoid: Do not invoke this prompt on every turn; it is expensive and adds latency. Do not evict items without logging; you will lose the ability to debug context-loss bugs. Do not skip the validation gate; a hallucinated item ID can crash your context manager. Finally, always pair this prompt with a checkpoint-and-resume strategy for tasks that cannot tolerate any context loss—eviction is a best-effort optimization, not a guarantee of perfect continuity.