Context Window Overflow Diagnosis Prompt

This prompt is for RAG developers and infrastructure engineers who need to diagnose a single production trace where the model's output suggests critical context was lost. The job-to-be-done is not general debugging; it is a focused forensic analysis to determine if a context window overflow caused information loss, which specific documents or chunks were dropped, and whether the packing strategy is to blame. Use this when a user reports an incomplete answer, a citation is missing, or an eval shows poor grounding, and you suspect the model never saw the necessary evidence.

The ideal user has access to a raw trace containing the final assembled prompt, the list of retrieved documents with their metadata, and the model's output. You should not use this prompt for latency or cost analysis, for multi-turn state corruption, or for hallucination diagnosis where the evidence was present but ignored. This prompt is specifically scoped to the mechanical failure mode of truncation: the evidence was retrieved but did not fit in the context window. Before running this prompt, confirm that the trace includes the pre-assembly retrieval list and the post-assembly prompt text so the analysis can compare what was available against what was actually sent.

After running this prompt, you will receive a context budget breakdown, a list of dropped or truncated documents, and a root-cause classification pointing to the packing strategy, token limit, or system prompt bloat. The next step is to feed this output into a validation harness that checks whether the overflow detection logic is correct—comparing the prompt's claimed dropped documents against a programmatic token-count simulation. Do not treat this prompt's output as the final word; always verify its findings against the raw token counts and the model's actual context limit before adjusting your chunk size, top-k, or packing order.

This prompt template is designed to analyze a single production trace and determine whether critical context was truncated due to context window limits. It requires the raw trace data, the context packing strategy used, and the expected output schema. The prompt instructs the model to reconstruct the context budget, identify dropped documents, and assess information loss. Use this template as a starting point for building an automated overflow detection harness in your observability pipeline.

textCopy
You are an AI observability engineer analyzing a single production trace for context window overflow.

Your task is to determine if the model's context window truncated critical information, which documents were dropped, and whether the packing strategy caused information loss.

## INPUTS

### Trace Data
[TRACE_DATA]

### Packing Strategy
[PACKING_STRATEGY]

### Model Context Window Limit (tokens)
[CONTEXT_WINDOW_LIMIT]

### Output Schema
[OUTPUT_SCHEMA]

## INSTRUCTIONS

1. Parse the trace data to extract:
   - The system prompt and its token count.
   - The user input and its token count.
   - All retrieved documents or context chunks, each with its token count and position in the packed context.
   - The final packed context sent to the model.
   - The model's output and any tool calls.

2. Reconstruct the context budget:
   - Calculate total tokens consumed by the system prompt, user input, and each document.
   - Identify the total tokens available after fixed overhead (system prompt + user input).
   - Determine which documents fit within the limit and which were truncated or dropped.

3. Analyze the packing strategy described in [PACKING_STRATEGY]:
   - Was the strategy followed correctly in this trace?
   - Did the strategy prioritize the most relevant documents?
   - Were any documents truncated mid-content, or were they dropped entirely?

4. Assess information loss:
   - Compare the model's output against the full set of retrieved documents (including dropped ones).
   - Flag any claims in the output that rely on information from dropped or truncated documents.
   - Rate the severity of information loss: NONE, MINOR, MODERATE, or CRITICAL.

5. Output your analysis in the exact schema specified in [OUTPUT_SCHEMA].

## CONSTRAINTS

- Do not hallucinate document content. Only reference documents explicitly present in the trace.
- If the trace data is incomplete or missing spans, flag it as INCOMPLETE_TRACE and explain what is missing.
- If no overflow occurred, state that clearly and provide the evidence.
- Use exact token counts from the trace where available. If token counts are missing, estimate using the formula: tokens ≈ words * 1.3, and flag the estimate as APPROXIMATE.

To adapt this template, replace each square-bracket placeholder with your actual data. For [TRACE_DATA], provide the raw trace in a structured format such as JSON or a serialized span list. For [PACKING_STRATEGY], describe the algorithm used to select and order documents—for example, 'top-10 by vector similarity score, concatenated in descending order.' For [CONTEXT_WINDOW_LIMIT], specify the model's maximum context length in tokens. For [OUTPUT_SCHEMA], define the exact JSON schema you expect, including fields for overflow_detected, dropped_documents, truncated_documents, information_loss_severity, and budget_breakdown. Before deploying this prompt in production, validate its output against a set of known overflow and non-overflow traces to ensure the model correctly identifies truncation events and does not produce false positives when the context window is sufficient.

This prompt is designed to be invoked programmatically as part of a trace analysis harness, not as a one-off chat interaction. The primary integration point is a post-retrieval, pre-generation hook or a batch trace review job that processes spans from your observability store (e.g., LangSmith, Arize, or a custom trace database). The harness must extract the required inputs—the final assembled prompt sent to the model, the list of retrieved documents with their metadata, and the model's output—from the raw trace data before calling this diagnosis prompt. Because the prompt performs structured analysis, you should invoke it with a model that supports strict JSON mode or function calling to guarantee the output schema is respected without repair loops.

The implementation should follow a validate-extract-diagnose pattern. First, validate that the trace contains all required fields: the full assembled_prompt, the retrieved_documents array with document_id, content, and position fields, and the model_output. If the trace is incomplete, the harness should log a trace_incomplete error and skip diagnosis rather than hallucinating missing data. Second, extract the context window budget by counting tokens in each section of the assembled prompt (system prompt, user input, retrieved documents, tool definitions) using the same tokenizer the production model uses. Third, call the diagnosis prompt with the extracted data and parse the JSON output. Wrap the call in a retry loop with a maximum of 2 attempts; if the model fails to return valid JSON matching the output schema, log a diagnosis_format_failure and fall back to a manual review flag. Store the parsed diagnosis result as a new span or annotation on the original trace for downstream alerting and dashboards.

For production deployments, integrate this prompt into your continuous monitoring pipeline by running it on a sample of traces that exceed a context-window utilization threshold (e.g., >90% of the model's context limit). Do not run it on every trace—the diagnosis prompt itself consumes tokens and adds latency. Instead, use a lightweight pre-filter: if context_utilization_pct < 0.85, skip diagnosis. If utilization exceeds the threshold, invoke the diagnosis prompt asynchronously and write the overflow_detected and critical_documents_truncated fields to your metrics store. Set an alert that fires when overflow_detected is true and critical_documents_truncated contains document IDs that match a high-priority knowledge base (e.g., compliance documents or safety policies). This targeted alerting prevents alert fatigue while catching the truncation events that carry the highest product risk.