Context Window Truncation Impact on Eval Prompt

This prompt is for RAG developers and AI reliability engineers who need to diagnose why a production response failed an automated evaluation when the root cause is suspected to be evidence truncated from the context window. The job-to-be-done is isolating whether the model had the right evidence but lost it during context assembly, versus never retrieving it at all. Use this prompt when you have a trace that captures the full retrieved document set, the final context window payload, the model output, and the eval scores. The prompt produces a before-and-after analysis showing exactly which documents or passages were dropped, their relevance scores, which eval dimensions were impacted, and a token budget attribution breakdown.

The ideal user has access to structured trace data—typically from an observability platform or logging pipeline—that includes per-step retrieval results with relevance scores, the assembled context payload sent to the model, and the eval rubric with dimension-level scores. Without this trace fidelity, the prompt cannot distinguish between a retrieval gap (the document was never fetched) and a truncation gap (the document was fetched but dropped to fit the context window). The prompt requires the following inputs as square-bracket placeholders: [RETRIEVED_DOCUMENTS] with relevance scores, [CONTEXT_WINDOW_PAYLOAD] showing what was actually passed to the model, [MODEL_OUTPUT], [EVAL_SCORES] with dimension-level breakdowns, and [TOKEN_BUDGET] showing the model's context limit and actual usage. Optional inputs include [RANKING_CONFIG] if a reranker was used and [TRUNCATION_STRATEGY] if the system applies a specific truncation policy.

Do not use this prompt when you lack trace data showing what was retrieved versus what was actually passed to the model. If the eval failure is clearly caused by retrieval gaps—the search step never returned relevant documents—use the Retrieval Gap Root-Cause for Eval Failure Prompt instead. Similarly, if the failure is caused by the model ignoring evidence that was present in the context window, this is a grounding or instruction-following problem, not a truncation problem, and requires a different diagnostic prompt. This prompt is specifically scoped to the narrow failure mode where evidence was present in the retrieval set but absent from the context window payload, and you need to quantify the eval impact of that absence.

This prompt template is designed to be pasted directly into your trace analysis tool, evaluation pipeline, or manual review workflow. It instructs the model to act as a forensic RAG auditor, comparing the original retrieval results against what actually fit inside the context window for a specific production request. The goal is to produce a structured, evidence-backed report that links truncation events to specific eval dimension failures, enabling you to decide whether to increase context budgets, refactor your prompt's assembly logic, or adjust your retrieval strategy.

textCopy
You are a forensic RAG auditor. Your task is to analyze a production trace where an LLM response failed an automated evaluation. Your specific focus is on context window truncation as the root cause.

## INPUT DATA
- **Original Retrieved Documents:** A list of all documents and their metadata returned by the retrieval system before context assembly.
  [RETRIEVED_DOCUMENTS]
- **Final Context Window Payload:** The exact text that was sent to the LLM as context.
  [FINAL_CONTEXT_PAYLOAD]
- **LLM Output:** The final response generated by the model.
  [LLM_OUTPUT]
- **Eval Failure Details:** The specific evaluation rubric, the failing score, and the evaluator's justification.
  [EVAL_FAILURE_DETAILS]
- **Token Budget:** The maximum token limit for the context window.
  [TOKEN_BUDGET]

## ANALYSIS INSTRUCTIONS
1.  **Identify Truncated Documents:** Compare [RETRIEVED_DOCUMENTS] against [FINAL_CONTEXT_PAYLOAD]. List every document or document chunk that was fully or partially excluded from the final context window.
2.  **Score Relevance:** For each truncated document, assign a relevance score (High, Medium, Low) to the user's implied query, based on its content and metadata.
3.  **Map to Eval Failure:** For each truncated document with High or Medium relevance, explain how its absence could have directly caused the specific failure described in [EVAL_FAILURE_DETAILS]. Link the missing information to a failed dimension in the rubric.
4.  **Attribute Token Spend:** Provide a breakdown of how the [TOKEN_BUDGET] was consumed. Categorize the spend into: System Prompt, Retrieved Documents (by ID), Conversation History, and Generated Output.
5.  **Formulate a Root-Cause Verdict:** State clearly whether context window truncation is the primary, contributing, or unlikely root cause of the eval failure.

## OUTPUT FORMAT
Return your analysis as a JSON object conforming to this schema:
{
  "truncation_analysis": [
    {
      "document_id": "string",
      "relevance_score": "High | Medium | Low",
      "truncation_type": "Full | Partial",
      "eval_impact": "string explaining the causal link to the eval failure, or null if relevance is Low"
    }
  ],
  "token_attribution": {
    "system_prompt_tokens": "number",
    "retrieved_document_tokens": "number",
    "conversation_history_tokens": "number",
    "generated_output_tokens": "number",
    "total_tokens_used": "number"
  },
  "root_cause_verdict": "Primary | Contributing | Unlikely",
  "verdict_rationale": "string"
}

## CONSTRAINTS
- Base your analysis strictly on the provided [RETRIEVED_DOCUMENTS] and [FINAL_CONTEXT_PAYLOAD]. Do not infer missing data.
- If the eval failure is unrelated to truncation, state that clearly in your verdict and rationale.
- Do not suggest fixes; focus only on diagnosis.

To adapt this template, replace the square-bracket placeholders with data from your production trace system. The [RETRIEVED_DOCUMENTS] should be a serialized list of objects, each with an id and content field at minimum. The [FINAL_CONTEXT_PAYLOAD] is the fully assembled string sent to the model. Ensure your trace tooling captures this pre- and post-assembly state, as it is the single most critical data dependency for this analysis. If your eval rubric has many dimensions, you can simplify [EVAL_FAILURE_DETAILS] to just the name of the failed dimension and the evaluator's one-line justification to keep the prompt focused.

This prompt is designed to be triggered programmatically when an eval failure correlates with a context window overflow event in your trace data. The implementation harness should listen for traces where context_window.truncated == true and the final response failed a quality eval. When both conditions are met, the harness extracts the full document list, the truncated document list, the final output, and the eval rubric, then invokes this prompt to produce a structured impact analysis. Do not run this prompt on every request—it is a diagnostic tool for failure investigation, not a real-time guardrail.

Wire the prompt into an investigation queue or a retryable job runner. The harness must: (1) validate that the trace contains both pre_truncation_documents and post_truncation_documents arrays with document_id, content, and relevance_score fields; (2) confirm that the eval failure dimensions are explicitly tagged in the trace metadata; (3) call the prompt with a model that supports the full context window needed to hold the document list plus the analysis instructions—Claude 3.5 Sonnet or GPT-4o are suitable defaults; (4) parse the output against the expected JSON schema and retry once if parsing fails; (5) log the analysis result back to the trace for future correlation. For high-stakes domains such as healthcare or legal review, add a human approval step before the analysis is considered final. The token budget attribution breakdown produced by the prompt should be compared against your actual token usage logs to validate the model's self-reported allocation.

Avoid wiring this prompt directly into an automated rollback or retry loop. The output is a diagnostic artifact, not a remediation action. After the analysis is produced, route it to the prompt engineer or RAG developer responsible for the retrieval pipeline. Common next steps include adjusting the context window budget allocation, re-ranking documents with stricter relevance thresholds, or implementing a summarization step for lower-ranked evidence. If the analysis reveals that critical evidence was truncated despite a high relevance score, prioritize a retrieval pipeline fix over a prompt patch.