Context Drift Detection Prompt for Chat Logs

This prompt is designed for monitoring pipelines that need to detect gradual topic or intent drift in long, multi-turn conversations. It is not a single-turn quality check. Use it when you need to identify conversations where the assistant follows tangents that lose the original user goal. The prompt produces drift scores per turn window and identifies the exact turn where the conversation diverged from the primary objective. This is a model-graded evaluation prompt, meaning it uses an LLM as a judge to analyze conversation logs. It belongs in your evaluation and observability stack, not in the real-time assistant loop. Run it asynchronously on sampled or flagged conversations to surface systemic drift patterns across your chat product.

The ideal user is an AI engineer or evaluation lead responsible for monitoring production chat quality. You need access to full conversation logs, including user messages, assistant responses, and any system prompts or tool outputs that shaped the interaction. The prompt requires a clearly defined primary objective for each conversation—this can be extracted from the initial user message, a session metadata tag, or a separate intent classifier. Without a well-defined objective, the drift detection becomes unreliable because there is no anchor to measure divergence against. You should also configure a drift threshold that matches your product's tolerance for tangents: a customer support chatbot may allow zero drift, while an open-ended creative assistant might accept moderate exploration before flagging.

Do not use this prompt for real-time guardrails or live intervention. It is an offline evaluation tool, not a production gate. Do not use it on conversations shorter than four turns, as drift patterns require sufficient interaction history to become detectable. Do not use it as a replacement for human review when conversations involve high-stakes decisions—drift scores should inform sampling strategies for human auditors, not replace them. If your assistant is designed to handle multiple intents within a single session, this prompt may produce false positives on legitimate topic shifts; in that case, pair it with a multi-intent tracking evaluation instead. Start by running this prompt on a labeled sample where you already know which conversations drifted, calibrate the threshold against your human judgments, and only then scale it across your production logs.

The prompt below is designed to analyze a multi-turn conversation log and identify where the assistant's responses begin to drift away from the user's original stated goal. It operates on a configurable sliding window of turns, producing a drift score for each window and flagging specific turns where problematic tangents begin. The template uses square-bracket placeholders for all variable inputs—replace these with your actual data before running the evaluation. The prompt is structured to return a strict JSON schema, making it suitable for automated pipelines rather than ad-hoc manual review.

textCopy
You are an expert conversation quality auditor. Your task is to analyze a multi-turn chat log and detect context drift—gradual or sudden shifts where the assistant's responses move away from the user's original stated goal or intent.

## INPUT

[CHAT_LOG]

## PARAMETERS

- Window size: [WINDOW_SIZE] turns
- Drift threshold: [DRIFT_THRESHOLD] (0.0 to 1.0, where higher values indicate more drift)
- Original user goal (explicitly stated or inferred from first user message): [USER_GOAL]

## INSTRUCTIONS

1. Extract the user's primary goal from the first user message. If [USER_GOAL] is provided, use it as the reference goal.
2. For each sliding window of [WINDOW_SIZE] consecutive assistant turns, compute a drift score from 0.0 (perfectly aligned with original goal) to 1.0 (completely unrelated to original goal).
3. For each window where the drift score exceeds [DRIFT_THRESHOLD], identify the specific turn index where drift began.
4. Classify each drift event into one of these categories:
   - `tangent_following`: Assistant followed a user-initiated tangent too far
   - `goal_substitution`: Assistant silently replaced the original goal with a different one
   - `overcorrection`: Assistant overcorrected after a user clarification, losing the original thread
   - `context_decay`: Assistant gradually forgot the original goal without any triggering event
   - `premature_closure`: Assistant treated the goal as resolved when it wasn't
5. For each drift event, provide a brief explanation citing the specific turns and messages that demonstrate the drift.

## OUTPUT SCHEMA

Return a JSON object with this exact structure:

{
  "original_goal": "string summarizing the extracted or provided user goal",
  "total_turns": number,
  "windows_analyzed": number,
  "drift_events": [
    {
      "window_start_turn": number,
      "window_end_turn": number,
      "drift_score": number,
      "drift_category": "tangent_following" | "goal_substitution" | "overcorrection" | "context_decay" | "premature_closure",
      "drift_start_turn": number,
      "explanation": "string citing specific messages and turn indices",
      "affected_turns": [number, ...]
    }
  ],
  "session_drift_trend": "stable" | "gradual_drift" | "sudden_drift" | "recovered" | "oscillating",
  "overall_drift_score": number,
  "recommendation": "string with actionable guidance for prompt or system improvement"
}

## CONSTRAINTS

- Do not flag legitimate topic shifts that the user explicitly requested.
- Do not penalize the assistant for asking clarifying questions that stay within scope of the original goal.
- If the user explicitly abandons the original goal and states a new one, treat the new goal as the reference from that point forward and note the goal change in the output.
- If [WINDOW_SIZE] is larger than the total number of assistant turns, analyze the entire conversation as a single window.
- Cite specific turn indices and message excerpts in every explanation.

To adapt this template, replace the square-bracket placeholders with your actual data. The [CHAT_LOG] should contain the full conversation with turn indices and speaker labels (User/Assistant). The [WINDOW_SIZE] controls sensitivity—smaller windows (2-3 turns) catch rapid drift, while larger windows (5-7 turns) identify gradual decay patterns. The [DRIFT_THRESHOLD] should be calibrated against your application's tolerance: support chatbots might accept 0.3-0.4 drift before intervention, while medical or legal assistants may require thresholds as low as 0.15. The [USER_GOAL] placeholder can be left empty if you want the model to infer the goal from the first user message, but providing it explicitly improves consistency when running this prompt across many conversations with varying opening styles.

Before deploying this prompt in production, run it against a golden dataset of conversations with known drift points and compare the model's drift scores against human-annotated ground truth. Pay special attention to false positives on legitimate topic shifts and false negatives on subtle goal substitution. If your use case involves high-stakes conversations where missed drift could cause harm, always route drift events above a configurable severity threshold for human review rather than relying solely on automated scoring.

The Context Drift Detection Prompt is designed to operate as a batch evaluation step within a chat monitoring pipeline, not as a real-time interceptor. After a conversation session closes or reaches a configurable turn threshold (e.g., every 10 turns), the full transcript is passed to the prompt. The primary integration point is an evaluation harness that feeds the transcript and the original user goal into the prompt, parses the structured drift scores, and writes the results to an observability store. This allows product teams to track drift trends over time without adding latency to the user-facing chat loop.

The harness must enforce a strict output schema. Expect a JSON response containing a drift_scores array, where each element maps to a turn window and includes window_start_turn, window_end_turn, drift_score (0.0 to 1.0), and a drift_explanation string. Implement a post-processing validator that checks for schema completeness, score range compliance, and window contiguity. If the model returns malformed JSON or missing windows, retry once with a repair prompt that includes the raw output and the schema. After validation, compare each window's drift_score against a configurable threshold (e.g., 0.7). Windows exceeding the threshold should trigger a drift alert that includes the turn range and explanation, routed to a human review queue or a dashboard for the conversation quality team.

Model choice matters for cost and reliability. This prompt requires strong instruction-following and structured output discipline, making it well-suited for GPT-4o, Claude 3.5 Sonnet, or equivalent models. Avoid smaller models that may struggle with the multi-window analysis and strict JSON format. For high-volume pipelines, batch multiple conversation transcripts into a single request where context windows permit, but ensure each transcript is clearly delimited to prevent cross-session contamination. Log every evaluation result—including raw model output, validation errors, and final scores—to enable traceability and future judge calibration. Do not use this prompt for real-time intervention; its role is offline monitoring and trend detection. The next step is to configure your alerting thresholds based on historical drift patterns and to establish a regular review cadence for flagged conversations.