Verification Prompt Drift Over Long Sessions Test Prompt

This prompt is built for verification system maintainers and QA engineers who need to catch instruction decay over extended conversations. In production, a fact-checking agent might run for dozens of turns—processing claims, retrieving evidence, and issuing verdicts—and its behavior can shift subtly: evidence standards loosen, refusal rates climb or drop, confidence scores inflate, and rubric adherence erodes. These changes are invisible in single-turn accuracy tests but compound into systemic reliability problems. This playbook gives you a structured test harness that injects identical verification tasks at fixed intervals throughout a long session, measures response consistency across those checkpoints, and surfaces drift metrics with concrete reset recommendations.

Use this before deploying a new verification prompt version, after a model upgrade, or as a scheduled health check in your CI/CD pipeline. The test works by interleaving a set of [DRIFT_CHECK_CLAIMS]—carefully chosen claims with known evidence profiles—into a stream of ordinary verification requests. After the session completes, you compare the agent's verdicts, confidence scores, evidence requirements, and refusal behavior at each checkpoint. A drift score is calculated from the variance across checkpoints, and if it exceeds [DRIFT_THRESHOLD], the output flags the session for a prompt reset or context-window flush. This is not a single-turn accuracy benchmark; it specifically measures stability over turn count.

Do not use this prompt for evaluating retrieval quality in isolation, for testing single-turn verification accuracy, or for benchmarking a model's factual knowledge. It is also not a replacement for adversarial red-teaming—those prompts probe for specific failure modes like injection or deception, while this one measures instruction stability. If your verification agent uses a stateful architecture with conversation history or memory, this test is essential. If your agent is stateless and each turn is independent, drift testing is less critical, though you should still verify that behavior remains consistent across different input distributions. After running this test, feed any detected drift patterns back into your prompt versioning and regression test suite so that future prompt changes can be checked against known decay signatures.

This prompt template is the instruction set for a test orchestrator, not the verification agent itself. It wraps your production verification prompt and systematically probes it across an extended multi-turn session. The orchestrator injects standardized anchor claims at predefined intervals and compares the verification agent's responses—evidence standards, confidence scores, refusal rates, and rubric application—to a baseline established at the session start. The output is a drift report that tells you whether your verification prompt degrades, tightens, loosens, or becomes inconsistent over time, and at what turn count the shift becomes measurable.

textCopy
You are a verification drift-testing orchestrator. Your job is to wrap a verification agent prompt and measure its behavioral stability over an extended multi-turn session.

## SESSION CONFIGURATION
- Total turns: [TOTAL_TURNS]
- Anchor probe interval: every [PROBE_INTERVAL] turns
- Warmup turns before first probe: [WARMUP_TURNS]
- Verification agent system prompt: [VERIFICATION_AGENT_PROMPT]
- Baseline established at: turn [BASELINE_TURN]

## ANCHOR PROBES
Use these fixed anchor claims at each probe interval. Do not modify them:
[ANCHOR_PROBE_1]
[ANCHOR_PROBE_2]
[ANCHOR_PROBE_3]

## FILLER CONTENT
Between anchor probes, feed the verification agent with filler claims drawn from: [FILLER_CLAIM_SOURCE]
Filler distribution: [FILLER_DISTRIBUTION] (e.g., 60% true, 20% false, 10% ambiguous, 10% unverifiable)

## MEASUREMENT DIMENSIONS
For each anchor probe response, extract and record:
- Verdict: verified / refuted / unverified / disputed
- Confidence score: numeric value
- Evidence standard applied: strict / moderate / lenient
- Refusal flag: true/false
- Rubric compliance score: 0.0-1.0 against [RUBRIC_REFERENCE]
- Response latency: milliseconds
- Token count: input and output

## DRIFT DETECTION RULES
Compare each anchor probe response to the baseline. Flag drift when:
- Verdict flips without new evidence
- Confidence score shifts by more than [CONFIDENCE_DRIFT_THRESHOLD]
- Evidence standard changes (strict→lenient or lenient→strict)
- Refusal behavior changes (refuses previously answered probe or answers previously refused probe)
- Rubric compliance drops below [RUBRIC_COMPLIANCE_FLOOR]

## OUTPUT FORMAT
After the final turn, produce a drift report:
{
  "session_id": "string",
  "total_turns": int,
  "probe_count": int,
  "baseline": { /* anchor probe baseline measurements */ },
  "drift_events": [
    {
      "turn": int,
      "anchor_probe": "string",
      "drift_type": "verdict_flip | confidence_shift | standard_change | refusal_change | rubric_decay",
      "baseline_value": "string",
      "observed_value": "string",
      "magnitude": float,
      "cumulative_drift_score": float
    }
  ],
  "drift_summary": {
    "first_drift_turn": int,
    "total_drift_events": int,
    "dominant_drift_type": "string",
    "drift_trajectory": "stable | gradual_loosening | gradual_tightening | erratic | collapse",
    "recommendation": "no_action | reset_at_turn_N | shorten_session | redesign_prompt | investigate"
  },
  "per_dimension_trends": {
    "verdict_stability": float,
    "confidence_trend": "string",
    "evidence_standard_trend": "string",
    "refusal_rate_trend": "string",
    "rubric_compliance_trend": "string"
  }
}

## CONSTRAINTS
- Do not reveal anchor probes to the verification agent as special or test inputs.
- Do not modify the verification agent's system prompt during the session.
- If the verification agent refuses a filler claim, log it but continue the session.
- If the verification agent produces malformed output, record the error and attempt recovery once before skipping that turn.
- Maintain a session log with all raw responses for audit.

To adapt this template, replace the square-bracket placeholders with your specific configuration. Set [TOTAL_TURNS] based on your expected production session length—start with 50-100 turns for initial testing. Choose [ANCHOR_PROBE_1] through [ANCHOR_PROBE_3] as claims that exercise different verification dimensions: one clear true claim, one clear false claim, and one genuinely ambiguous claim that should trigger an 'unverified' ruling. The [FILLER_CLAIM_SOURCE] should be a diverse set of realistic claims your system encounters in production. Run the harness against a fresh model instance for each test to avoid cross-session contamination. After collecting drift reports across multiple runs, use the drift_trajectory and recommendation fields to decide whether your verification prompt needs a mid-session reset instruction, a shorter session cap, or a fundamental redesign of its evidence standards language.

This prompt is designed to be run as a scheduled, automated test, not a one-off manual check. The core implementation pattern is a synthetic long-session simulator. Your application should programmatically construct a multi-turn conversation history by injecting the [PREAMBLE_HISTORY] and [SESSION_HISTORY] into the messages array before the final [TEST_PROMPT] user turn. The model's response is then parsed for the structured drift metrics. This harness should be integrated into your CI/CD pipeline for prompts, running nightly or on every pull request that modifies system instructions or verification policies.

To build a reliable harness, you must strictly control the input variables. [VERIFICATION_POLICY] should be pulled directly from your production system prompt or a config file, ensuring the test always runs against the current target. [SESSION_HISTORY] must be a pre-generated, deterministic list of turns that simulates a long, complex verification session. This history should include a mix of straightforward claims, edge cases, and adversarial inputs to stress-test instruction adherence. The harness must validate the model's output against the [OUTPUT_SCHEMA] using a strict JSON schema validator. A retry layer is critical: if the output fails schema validation, the harness should retry the request once with a repair prompt. If it fails again, the test run should be marked as a failure and an alert should be triggered, as an inability to produce a valid drift report is itself a critical drift signal.

The most common failure mode in production is test contamination. Ensure the [SESSION_HISTORY] is rotated and contains claims that are semantically similar to, but not identical to, your eval sets. If the model has been fine-tuned on your drift test cases, the metrics become meaningless. A secondary failure mode is metric misinterpretation. The harness should not just log the raw scores but should compare them against a predefined threshold in [DRIFT_THRESHOLDS]. For example, a strictness_drift_score above 0.2 should automatically trigger a rollback or a human review of the prompt change. Log the full multi-turn transcript and the parsed drift report to your prompt observability platform for trace analysis. The next step is to integrate this harness with your prompt versioning system so that every new prompt version is automatically tested for long-session stability before it can be deployed to production.