Retry Budget for Context Window Overflow Recovery Prompt

This prompt is designed for prompt engineers and application developers who manage long-context workflows where a model's context window is a hard physical limit. The job-to-be-done is not just to recover from a single overflow, but to define a strict retry budget that governs how many compression, summarization, or chunking attempts are allowed before the system must stop retrying and escalate. The ideal user is someone integrating an LLM into a production pipeline—such as a document analysis service, a long-form report generator, or a multi-turn agent with persistent memory—where silently truncating context or entering an infinite compression loop is unacceptable. You need this prompt when you have already designed a primary context management strategy (e.g., a sliding window or RAG-based retrieval) and that strategy has failed because the combined prompt and response still exceed the model's maximum token limit.

You should not use this prompt as your primary context management strategy. It is a recovery mechanism, not a design pattern. If you are routinely hitting context window overflows, the root cause is likely in your prompt assembly or retrieval pipeline, and you should invest in better context budgeting or compression before relying on this retry loop. This prompt is also inappropriate for real-time, latency-sensitive applications where multiple retries would breach a strict SLA; in those cases, a single-attempt fallback to a summary-based response or an immediate human handoff is preferred. The prompt assumes you have access to a reliable token-counting utility and a mechanism to track the state of the conversation or document across retries, as the core risk is information loss—critical facts, instructions, or user intent that are dropped during aggressive compression.

Before implementing this prompt, ensure you have instrumented your application to detect the specific context_length_exceeded error from your model provider and can pass the original, full context and the failed request into the recovery loop. The retry budget should be configured as a discrete integer (e.g., 3 attempts) and paired with a preservation validator that checks if key entities, constraints, and the user's original objective are still present in the compressed prompt after each attempt. The next step is to wire this prompt into a harness that counts attempts, compares pre- and post-compression outputs for semantic drift, and escalates to a predefined fallback—such as a human review queue or a simpler summary model—when the budget is exhausted.

This prompt template is designed to be inserted into a recovery harness that activates when a model response is truncated due to context window overflow. Instead of blindly retrying, the prompt instructs the model to attempt recovery within a defined budget of compression or chunking attempts. If the budget is exhausted, the model must produce a structured escalation payload rather than continuing to loop. The template uses square-bracket placeholders that your application must populate before sending the request.

textCopy
You are a context window overflow recovery agent. Your task is to complete a response that was truncated because the combined input and output exceeded the model's context window.

ORIGINAL_REQUEST: [ORIGINAL_USER_REQUEST]
TRUNCATED_RESPONSE: [TRUNCATED_OUTPUT]
REMAINING_CONTEXT_TO_PROCESS: [UNPROCESSED_CONTEXT]

RECOVERY_BUDGET:
- Maximum compression attempts: [MAX_COMPRESSION_ATTEMPTS]
- Maximum chunking attempts: [MAX_CHUNKING_ATTEMPTS]
- Attempts already consumed: [ATTEMPTS_CONSUMED]
- Budget exhaustion action: [ESCALATION_ACTION]

RECOVERY_RULES:
1. If attempts remain, choose ONE recovery strategy:
   a. COMPRESSION: Summarize [UNPROCESSED_CONTEXT] into a dense form that preserves all critical facts, entities, and relationships. Then generate the completion.
   b. CHUNKING: Split [UNPROCESSED_CONTEXT] into [CHUNK_SIZE] segments. Process the first segment now and indicate that [REMAINING_SEGMENTS_COUNT] segments remain.
2. If the budget is exhausted, do NOT attempt further recovery. Instead, produce an ESCALATION_PAYLOAD.
3. Never fabricate information not present in the provided context.
4. Preserve all citations and source references from the original context.

ESCALATION_PAYLOAD_FORMAT:
{
  "status": "budget_exhausted",
  "attempts_consumed": [ATTEMPTS_CONSUMED],
  "recovery_methods_tried": [LIST_OF_METHODS],
  "unprocessed_context_summary": "[BRIEF_SUMMARY]",
  "truncated_output_available": true,
  "recommended_fallback": "[SUMMARY_FALLBACK_OR_HUMAN_REVIEW]",
  "estimated_remaining_tokens": [ESTIMATED_TOKENS]
}

OUTPUT_REQUIREMENTS:
- If recovery succeeds: Output the completed response only, with no meta-commentary.
- If budget exhausted: Output ONLY the valid JSON escalation payload.
- Do not include both a completion and an escalation payload.

To adapt this template, your application harness must track [ATTEMPTS_CONSUMED] across retry cycles and inject the correct values for [MAX_COMPRESSION_ATTEMPTS] and [MAX_CHUNKING_ATTEMPTS]. The [ESCALATION_ACTION] placeholder should contain a concrete instruction such as 'route to human review queue' or 'generate summary fallback.' Before deploying, validate that your harness correctly parses the escalation JSON and routes it to the appropriate downstream system. For high-stakes workflows where information loss is unacceptable, configure the budget conservatively and ensure the escalation path includes human review with the full [UNPROCESSED_CONTEXT] attached.

The retry budget for context window overflow recovery is not a standalone prompt; it is a control-plane instruction that sits inside a retry loop managed by your application harness. The harness is responsible for detecting a context window overflow error (typically a 400 or 429 error with a token limit message, or a truncated response with a finish_reason of length), invoking the recovery prompt, tracking the retry budget, and deciding when to stop retrying and escalate. The prompt itself defines the compression or chunking strategy, but the harness enforces the budget, validates the output, and packages the escalation payload when the budget is exhausted.

Wire the prompt into a retry loop with the following components. Error detection: intercept model API responses and classify context_length_exceeded, token_limit_reached, or incomplete finish_reason=length as overflow events. Budget state: maintain a counter of recovery attempts and a configurable maximum (e.g., 3 attempts). Each attempt increments the counter and logs the compression strategy used, the token count before and after compression, and the model's response. Prompt invocation: on overflow, inject the original [INPUT], [CONTEXT], and [RETRY_HISTORY] into the recovery prompt template. The recovery prompt should return a compressed or chunked version of the context along with a continuation instruction. Output validation: before resubmitting, validate that the compressed context is well-formed, that key entities and constraints from the original [CONTEXT] are preserved, and that the total token count is under the model's limit. Use a tokenizer library (e.g., tiktoken for OpenAI models) to verify the count programmatically. Budget exhaustion: when the retry counter reaches the maximum, stop retrying. Package the [RETRY_HISTORY], the original [INPUT], the last compressed context, and the error trace into a structured escalation payload and route it to a summary-based fallback model, a human review queue, or a dead-letter topic.

For high-stakes workflows where information loss during compression could cause harm, add an information preservation eval inside the harness. After each compression attempt, run a lightweight check: extract a set of critical facts from the original context and the compressed context, and verify that all critical facts are present. This can be a separate LLM call with a strict rubric or a deterministic key-entity check. If the eval fails, treat it as a failed retry and increment the budget counter. Logging and observability: log every retry attempt with the attempt number, compression strategy, token counts, validation results, and the final escalation decision. These logs are essential for tuning the retry budget, diagnosing compression failures, and providing audit evidence for escalation reviewers. Avoid silent compression failures—if the harness cannot validate that the compressed context is faithful, escalate immediately rather than submitting degraded context to the downstream task.