Feedback-Driven Context Window Inspection Prompt

This prompt is for RAG developers and search engineers who need to audit exactly what was inside the model's context window when a user reported a quality problem. The job-to-be-done is forensic: a user has flagged a response as irrelevant, incomplete, or hallucinated, and you must determine whether the root cause is a retrieval gap, a context truncation issue, or a ranking failure. The prompt reconstructs the context window from trace data, scores each chunk for relevance to the user's query, and identifies what was missing or truncated. Use this when you have access to a production trace containing the retrieved documents, the final assembled context, and the user's feedback signal.

Do not use this prompt when you lack trace data that captures the exact context window contents at inference time. If your observability pipeline only logs final outputs without retrieval intermediates, you cannot perform a meaningful context-window audit. This prompt is also not a substitute for offline retrieval evaluation against a golden dataset—it is a reactive, single-session diagnostic tool, not a batch evaluation system. The prompt requires the user's original query, the feedback signal (e.g., 'irrelevant response,' 'missing information,' 'hallucination'), and the raw trace spans that include retrieved chunks, their ranking scores, and the final assembled context. Without these inputs, the audit will produce unreliable or incomplete results.

The output is a structured context-window audit containing: a list of chunks that were included in the context window with per-chunk relevance scores, a list of chunks that were retrieved but truncated or excluded, and a diagnostic summary explaining whether missing or irrelevant context explains the user's feedback. This audit is designed to be consumed by a human engineer making a fix decision—update the retrieval query, adjust chunk size, re-rank, or expand the context budget. Always validate the audit against the original trace spans to confirm that no chunks were misattributed. For high-severity user reports where the fix will be deployed to production, require a second engineer to review the audit before accepting the diagnostic conclusion.

This template is designed to be copied directly into your prompt library or observability tool. It instructs the model to act as a context-window auditor, reconstructing what the model saw during a specific request that received user feedback. The prompt forces the model to list every chunk, assess relevance, and identify whether missing or irrelevant context explains the reported issue. All dynamic inputs are represented as square-bracket placeholders that your application must populate before execution.

textCopy
You are a context-window auditor for a RAG system. Your job is to inspect the full context window from a production request that received user feedback and determine whether the context explains the reported quality issue.

## INPUTS

### User Feedback
[USER_FEEDBACK]

### User Query (Original)
[USER_QUERY]

### Model Response (As Delivered to User)
[MODEL_RESPONSE]

### Retrieved Context Chunks (In Order of Retrieval)
[RETRIEVED_CHUNKS]

### Truncation Metadata
- Max context window size: [MAX_TOKENS]
- Actual tokens used: [ACTUAL_TOKENS]
- Chunks truncated: [TRUNCATED_CHUNK_IDS]

### Retrieval Configuration
- Retrieval query used: [RETRIEVAL_QUERY]
- Retrieval method: [RETRIEVAL_METHOD]
- Top-K setting: [TOP_K]
- Reranker applied: [RERANKER_NAME]

## OUTPUT SCHEMA

Return a JSON object with this exact structure:
{
  "audit_summary": {
    "total_chunks_retrieved": number,
    "chunks_in_context_window": number,
    "chunks_truncated": number,
    "overall_relevance_score": number (0.0 to 1.0),
    "missing_context_likely": boolean,
    "irrelevant_context_likely": boolean,
    "truncation_impact": "none" | "low" | "medium" | "high"
  },
  "chunk_analysis": [
    {
      "chunk_id": string,
      "position_in_window": number,
      "relevance_score": number (0.0 to 1.0),
      "relevance_rationale": string (one sentence),
      "used_in_response": boolean,
      "evidence_in_response": string or null (exact quote from response if used),
      "truncated": boolean
    }
  ],
  "feedback_alignment": {
    "feedback_type": string (classify: "missing_info" | "incorrect_info" | "irrelevant_response" | "incomplete_response" | "other"),
    "context_explains_feedback": boolean,
    "explanation": string (how context does or does not explain the feedback),
    "root_cause_candidate": "retrieval_gap" | "ranking_failure" | "truncation" | "irrelevant_retrieval" | "prompt_issue" | "model_error" | "unclear"
  },
  "recommendations": [
    {
      "action": string,
      "target": "retrieval_query" | "top_k" | "reranker" | "chunk_size" | "context_window" | "prompt_instructions" | "model" | "investigate_further",
      "rationale": string (one sentence tied to audit evidence)
    }
  ]
}

## CONSTRAINTS
- Score relevance against the user query AND the user feedback, not against the model response.
- If a chunk was truncated, mark it and assess whether the truncated portion likely contained needed information.
- For the root_cause_candidate, select only ONE primary cause. Use "unclear" if evidence is insufficient.
- Every recommendation must reference specific chunk IDs or metadata from the audit.
- Do not speculate about retrieval system internals beyond what is provided in the metadata.

## RISK_LEVEL: [RISK_LEVEL]
- If RISK_LEVEL is "high": flag the audit for human review and do not auto-apply recommendations.
- If RISK_LEVEL is "medium": recommendations can be queued for review but not auto-deployed.
- If RISK_LEVEL is "low": recommendations can be logged for trend analysis.

Adaptation notes: Replace each square-bracket placeholder with data from your trace store or observability platform. The [RETRIEVED_CHUNKS] field should contain the full text of each chunk with its ID and retrieval rank. If your system does not capture truncation metadata, remove the truncated fields from the schema or set them to false. The [RISK_LEVEL] placeholder should be set by your application based on the severity of the user feedback—use "high" for complaints involving factual errors, safety concerns, or regulated content. For low-stakes feedback, you can remove the human-review constraints to enable automated analysis pipelines.

This prompt is designed to be triggered by a user feedback event, such as a 'thumbs down' or a support ticket flagging a low-quality answer. The implementation harness should automatically fetch the full trace for the flagged session, extract the final context window payload, and pass it to the prompt alongside the user's original query and the model's generated response. The goal is to produce a structured audit that your observability dashboard or support tool can display, allowing an engineer to immediately see which chunks were relevant, which were missing, and whether truncation explains the failure.

To wire this into an application, build a feedback webhook handler that queries your trace store (e.g., LangSmith, Arize, or a custom OpenTelemetry backend) for the retrieval_context and generation spans associated with the flagged session_id. Construct the [CONTEXT_WINDOW] variable by serializing the retrieved documents in order, preserving their doc_id, position, and content fields. The [USER_QUERY] and [MODEL_OUTPUT] variables come directly from the trace. Before calling the LLM, validate that the trace is complete and that the context window payload is not empty; if it is, return an error immediately rather than invoking the model. After receiving the LLM's JSON output, validate it against the expected schema—each chunk must have a relevance_score between 0 and 1, and the truncation_analysis must account for the model's actual context limit. Log the full audit result to your observability platform, linking it back to the original trace and feedback event for future querying.

For high-stakes applications where a missed retrieval gap could impact user trust or regulatory compliance, require a human to review the audit before closing the feedback loop. Implement a retry with exponential backoff if the LLM call fails or returns malformed JSON, but stop after three attempts and escalate to a manual review queue. Avoid running this prompt on every session—it is cost-effective only when triggered by explicit negative feedback or a sampling rule for quality assurance. Finally, store the audit results in a structured format that allows you to aggregate findings over time, identifying systemic retrieval issues like consistently low relevance scores for a specific document source or frequent truncation of long documents.