Example Drift Detection for A/B Test Variants Prompt

This prompt is designed for experimentation teams and MLOps engineers who are running A/B tests on prompt variants and need to ensure the validity of their results. Its primary job is to detect and flag when the few-shot examples used in a control prompt and a treatment prompt are drifting differently from each other relative to the incoming production data. This 'differential drift' is a critical confounder: if one variant's examples become stale faster than the other's, any measured performance difference might be due to example quality divergence rather than the prompt change being tested. The ideal user is someone who already has a live A/B test running, access to production traffic samples for each variant, and a way to action the resulting parity report.

You should use this prompt when you have a scheduled health check for an active experiment or when investigating an unexpected performance gap between variants. It requires several structured inputs: a representative sample of recent production inputs routed to each variant, the full set of few-shot examples for both the control and treatment prompts, and a configured drift detection method (e.g., embedding similarity threshold). The prompt is not a replacement for standard output evaluation or statistical significance tests; it is a diagnostic tool specifically for the example sets. Do not use this prompt before an experiment launches—it is for monitoring live tests, not for pre-launch validation. It is also inappropriate for experiments that do not use few-shot examples, as there are no example sets to compare.

The output is a structured drift parity report, not a simple pass/fail flag. It will identify specific examples in each variant that are drifting, quantify the magnitude of the drift asymmetry, and provide a risk assessment for the experiment's validity. The report should be wired into your experiment monitoring dashboard or CI/CD pipeline. If the prompt flags a high-severity differential drift, the next step is to pause the experiment, refresh both example sets from the same recent production distribution, and restart the test to ensure a fair comparison. Avoid the temptation to refresh only the underperforming variant's examples, as this introduces a new confounder.

This template is designed to be dropped directly into your experimentation pipeline. It accepts two sets of few-shot examples—one for the control variant and one for the treatment—alongside a sample of recent production inputs. The prompt instructs the model to analyze whether the examples in each variant are drifting differently from the production distribution, which would confound your A/B test results. Use this when you need a structured, repeatable method to ensure that measured performance differences between variants are due to prompt changes, not example staleness.

codeCopy
You are an AI experimentation auditor. Your task is to detect whether the few-shot example sets used in two A/B test variants are drifting differently from the current production input distribution.

## CONTROL VARIANT EXAMPLES
[CONTROL_EXAMPLES]

## TREATMENT VARIANT EXAMPLES
[TREATMENT_EXAMPLES]

## PRODUCTION INPUT SAMPLE
[PRODUCTION_INPUTS]

## INSTRUCTIONS
1. Analyze the production input sample to identify its primary topics, intents, difficulty levels, and vocabulary patterns.
2. For each variant's example set, assess how well it represents the production distribution. Consider:
   - Topic coverage and gaps
   - Vocabulary and terminology alignment
   - Complexity and difficulty match
   - Format and structure consistency
3. Compare the drift patterns between the control and treatment example sets. Determine if one variant's examples are significantly more stale, misaligned, or unrepresentative than the other.
4. Classify the drift parity into one of these categories:
   - **No significant drift difference**: Both variants are similarly aligned with production.
   - **Moderate drift asymmetry**: One variant shows noticeably more drift; results may be partially confounded.
   - **Critical drift asymmetry**: One variant's examples are severely misaligned; experiment results are likely invalid.
5. If drift asymmetry is detected, identify which specific examples in the worse-performing variant are contributing most to the divergence.

## OUTPUT SCHEMA
Return a JSON object with this exact structure:
{
  "drift_parity": "no_difference|moderate_asymmetry|critical_asymmetry",
  "control_drift_score": 0.0-1.0,
  "treatment_drift_score": 0.0-1.0,
  "drift_delta": float,
  "affected_variant": "control|treatment|both|neither",
  "asymmetry_explanation": "string",
  "stale_example_indices": ["control_example_3", "treatment_example_7"],
  "recommended_action": "continue_experiment|pause_and_refresh|discard_results",
  "confidence": 0.0-1.0
}

## CONSTRAINTS
- Base all assessments on observable distribution differences, not speculation.
- If production sample size is too small for reliable comparison, set confidence low and note this in the explanation.
- Do not evaluate the quality of the prompt instructions themselves—only the example sets.
- Flag any examples that reference outdated product features, deprecated APIs, or obsolete terminology.

To adapt this template, replace the square-bracket placeholders with your actual data. [CONTROL_EXAMPLES] and [TREATMENT_EXAMPLES] should contain the full few-shot example blocks used in each variant, formatted consistently. [PRODUCTION_INPUTS] should be a representative sample of recent user inputs from the experiment's traffic window. If your production sample exceeds the model's context window, preprocess it by clustering inputs and providing representative samples per cluster. Before deploying, validate that the output JSON parses correctly and that drift scores correlate with manual spot checks. For high-stakes experiments where invalid results carry business risk, route critical_asymmetry outputs for human review before taking action on the experiment.

This prompt is designed to be called programmatically as part of an automated A/B test health-check pipeline, not as a one-off manual query. The implementation harness should run the prompt on a scheduled cadence (e.g., daily or per test-phase gate) and treat the output as a structured signal for experiment validity. The primary integration points are your experimentation platform's logging system, your prompt management or model gateway, and your alerting or incident management tool. The harness is responsible for assembling the required inputs, invoking the model, validating the structured output, and routing the result to the appropriate downstream system.

To wire this into an application, start by constructing the [CONTROL_EXAMPLES] and [TREATMENT_EXAMPLES] arrays from your experiment configuration store. Each example should be a serialized object containing the input, the expected output, and metadata such as the example's creation date and intended use case. The [PRODUCTION_SAMPLE] should be pulled from your logging pipeline, sampling recent production inputs from users assigned to each variant. Use a consistent sampling window (e.g., the last 7 days) and a minimum sample size (e.g., 500 inputs per variant) to ensure statistical stability. Before calling the model, validate that the sample sizes are sufficient and that the example sets are non-empty; if not, abort the run and log a data-quality error rather than sending a malformed prompt. The model should be called with response_format set to a strict JSON schema that matches the expected DriftParityReport structure, and the temperature should be set to 0 to maximize reproducibility. Implement a retry loop with exponential backoff (up to 3 attempts) for transient API failures, but do not retry on validation failures—those indicate a prompt or data problem that requires human investigation.

After receiving the model response, validate the output against the expected schema before acting on it. Check that drift_parity_score is a float between 0 and 1, that divergent_example_indices contains valid indices within the provided example sets, and that confounding_risk_level is one of the allowed enum values. If validation fails, log the raw response and the validation error, then escalate to the prompt engineering team for debugging. If validation passes, write the full report to your experiment monitoring dashboard and evaluate the confounding_risk_level. For high risk, automatically trigger an alert to the experimentation team with the recommended_action and evidence_summary fields. For medium risk, create a ticket for review within the next business day. For low risk, simply log the result for audit purposes. Avoid the mistake of treating this prompt's output as a hard gate that automatically stops an experiment; it should inform a human decision about whether the experiment results remain trustworthy.