Context Window Overflow Root-Cause Prompt

This prompt is designed for platform engineers and AI SREs who need to diagnose a production incident where a model request failed due to exceeding the context window limit. The job-to-be-done is not just to confirm an overflow occurred, but to perform a structured root-cause analysis on a single production trace. The ideal user has access to the raw trace data, including the final token counts, the assembled prompt, and ideally the sequence of tool calls or retrieval steps that preceded the error. The required context is a complete trace log, not just the error message, because the goal is to attribute the overflow to a specific component: the system prompt, retrieved documents, conversation history, or an unbounded tool output.

This prompt should not be used for aggregate cost analysis or capacity planning across many requests; it is a surgical diagnostic tool for a single failed trace. It is also inappropriate for real-time prevention. Do not wire this prompt into the hot path of a request to decide if the next call will fail. Instead, use it offline during post-incident review or as part of a debugging workflow triggered by a spike in 400 errors from a model provider. The prompt assumes the overflow has already happened and that the trace is available for inspection. It is not a replacement for programmatic token budgeting, which should be implemented in the application layer using a tokenizer library before the request is sent.

Before using this prompt, ensure you have sanitized the trace of any sensitive customer data, as the analysis requires inspecting the full prompt contents. The output is a structured root-cause classification and a remediation recommendation, which you should validate against your own understanding of the system architecture. If the prompt classifies the overflow as 'preventable,' the next step is to implement the recommended fix in your context assembly code, not to repeatedly run this prompt on new failures. If the classification is 'unpreventable,' the recommendation should guide a product decision, such as increasing the context budget or redesigning the feature to use fewer tokens.

The following prompt is designed to be copied directly into your observability or debugging harness. It takes a structured production trace and a token limit as input, then produces a root-cause classification and a remediation recommendation. The prompt uses square-bracket placeholders for all dynamic inputs, making it safe to template into your application code, a CI/CD pipeline, or an internal debugging notebook.

textCopy
You are an AI platform engineer diagnosing a context window overflow in a production trace. Your goal is to identify the root cause and recommend a specific, actionable fix.

## INPUT
- Full Request Trace (JSON): [TRACE_JSON]
- Model Context Window Limit (tokens): [TOKEN_LIMIT]

## ANALYSIS STEPS
1. Parse the trace to identify every component that consumed tokens: system prompt, user message, conversation history, retrieved documents, tool definitions, tool call results, and any other injected context.
2. Calculate the token count for each component. If exact counts are not in the trace, estimate using the component's character length divided by 4.
3. Identify which single component, when added, caused the total token count to exceed [TOKEN_LIMIT].
4. Determine whether the overflow was preventable (e.g., a runaway conversation loop, an overly broad retrieval query) or unavoidable (e.g., a single document that is larger than the context window).

## OUTPUT SCHEMA
Return a JSON object with this exact structure:
{
  "total_tokens_used": <number>,
  "overflow_amount": <number>,
  "root_cause_component": "system_prompt" | "user_input" | "conversation_history" | "retrieved_documents" | "tool_definitions" | "tool_results" | "other",
  "root_cause_detail": "<specific description of the offending content, e.g., 'Retrieved document #3: 8,500 tokens'>",
  "preventable": <boolean>,
  "remediation": "<one-sentence, actionable recommendation>"
}

## CONSTRAINTS
- If the trace is malformed or missing critical fields, set root_cause_component to "other" and explain in root_cause_detail.
- Do not hallucinate token counts. Use only data present in [TRACE_JSON].
- The remediation must be specific and implementable (e.g., 'Reduce max_retrieved_docs from 10 to 5' not 'Use less context').

To adapt this template, replace [TRACE_JSON] with a serialized trace object from your logging pipeline. The trace must include at minimum a breakdown of prompt components and their sizes. If your tracing system does not capture per-component token counts, you will need to add that instrumentation before this prompt can produce reliable results. For high-severity incidents, always pair the model's output with a human review step before applying the remediation to production configuration.

This prompt is designed to be called programmatically as part of an automated incident response or trace analysis pipeline. It should not be used in a free-text chat interface. The primary integration point is a webhook or event-driven function that triggers when a token limit error (HTTP 400, 413, or model-specific 'context_length_exceeded') is logged. The function retrieves the full trace payload from your observability store, hydrates the [TRACE_JSON] and [MODEL_TOKEN_LIMIT] placeholders, and sends the assembled prompt to a fast, instruction-following model like gpt-4o-mini or claude-3-haiku for cost-effective analysis. The output must be parsed as structured JSON and written back to the incident ticket or monitoring dashboard.

Validation is critical before the output is trusted. Implement a post-processing validator that checks the parsed JSON for the required fields: overflow_root_cause, overflow_category, responsible_component, was_preventable, and remediation_recommendation. If the overflow_category is not one of the expected enums (system_prompt, retrieved_context, tool_results, conversation_history, generated_output, fixed_overhead), discard the result and retry with a stricter [OUTPUT_SCHEMA] constraint. Log the raw model response and the validation failure for later prompt tuning. For high-severity incidents where the overflow caused a user-facing error, route the output to a human reviewer before the remediation recommendation is applied to the production prompt configuration.

Model choice matters for this workflow. Use a model with strong JSON mode or structured output support to minimize parsing failures. The prompt's [CONSTRAINTS] section should include explicit instructions to return only valid JSON without markdown fences. In your application harness, wrap the model call in a retry loop (max 3 attempts) with exponential backoff. On each retry, append the previous parsing error to the prompt as additional context: 'The previous response failed JSON validation with error: [ERROR_MESSAGE]. You MUST return only valid JSON that matches the schema.' This self-correction pattern significantly improves reliability in automated pipelines. For teams using LangChain or similar frameworks, this prompt can be wrapped as a custom tool that the incident-response agent calls, with the tool's output schema enforcing the expected fields.

Avoid wiring this prompt directly to an automated remediation action. The remediation_recommendation field may suggest changes to system prompts, retrieval chunk sizes, or conversation history trimming logic. These changes should always be staged as proposed diffs in a prompt version control system, not applied live without review. The prompt's was_preventable flag is a useful routing signal: if true, create a task in the engineering backlog to implement the recommended guardrail; if false, log the incident for capacity planning and model migration discussions. Finally, aggregate the overflow_category field across incidents to build a dashboard that shows which component most frequently causes token budget exhaustion, turning individual root-cause analyses into a systemic observability signal.