Semantic Drift Detection Prompt for Prompt Versions

This prompt is for prompt engineers and QA leads who need to quantify how a change to a system prompt, instruction, or few-shot example alters the model's behavior across a known set of inputs. The job-to-be-done is not simple string comparison; it is detecting whether the intent and meaning of the output have shifted in ways that break downstream contracts, violate safety policies, or degrade user experience. Use this when you have a stable golden dataset of input-output pairs and you are preparing to promote a new prompt version to staging or production. The ideal user has access to a regression test suite and needs a structured, machine-readable drift report to feed into an automated release gate.

Do not use this prompt for ad-hoc spot-checking a single example, for evaluating a brand-new prompt with no baseline, or when your test dataset is too small to produce statistically meaningful clusters. This prompt is also inappropriate when the change is purely cosmetic (e.g., fixing a typo in an instruction that does not alter model behavior) or when you need a binary pass/fail judgment without explanation. The prompt assumes you are comparing two stable prompt versions—[PROMPT_VERSION_A] and [PROMPT_VERSION_B]—against a shared [GOLDEN_DATASET] of at least 30 examples. If your dataset is smaller, the clustering and threshold recommendations will be unreliable.

Before using this prompt, ensure you have collected the required inputs: a JSON array of test cases with input and expected_output fields, the two prompt templates to compare, and any [DOMAIN_CONSTRAINTS] such as forbidden terminology or required disclosures. The output is a drift report, not a final release decision. You must still apply your own [PASS_FAIL_THRESHOLDS] for semantic similarity and intent preservation. If the prompt is used in a regulated domain (healthcare, finance, legal), a human reviewer must sign off on any clusters flagged as high-severity drift before the new prompt is promoted. After reading this section, proceed to the prompt template to copy the exact instructions, then review the implementation harness for wiring this into an automated CI/CD pipeline.

The following template is designed to be used as a meta-evaluator: it takes the outputs generated by two different prompt versions (A and B) for the same set of inputs and produces a structured drift report. You should run this prompt once per example in your golden dataset, or batch multiple examples together if your context window permits. The prompt expects you to provide the original input, the expected reference behavior, and the actual outputs from both versions.

textCopy
You are a semantic drift detector for prompt regression testing. Your job is to compare two outputs generated by different prompt versions for the same input and determine whether a meaningful behavioral change has occurred.

## INPUT
- Original User Input: [INPUT]
- Expected Behavior (from golden dataset): [EXPECTED_BEHAVIOR]
- Output from Prompt Version A (baseline): [OUTPUT_A]
- Output from Prompt Version B (candidate): [OUTPUT_B]

## CONSTRAINTS
- [CONSTRAINTS]

## OUTPUT SCHEMA
Return a JSON object with exactly these fields:
{
  "semantic_similarity_score": <float between 0.0 and 1.0, where 1.0 means identical meaning>,
  "intent_preserved": <boolean, true if Version B fulfills the same user intent as Version A>,
  "drift_detected": <boolean, true if a meaningful behavioral change occurred>,
  "drift_category": <string, one of: "none", "format_change", "tone_shift", "content_divergence", "refusal_change", "hallucination_introduced", "hallucination_removed", "completeness_change", "other">,
  "drift_severity": <string, one of: "none", "low", "medium", "high", "critical">,
  "drift_description": <string, concise explanation of what changed and why it matters>,
  "version_b_better": <boolean or null, true if Version B is an improvement, false if it's worse, null if neutral or unclear>,
  "requires_human_review": <boolean, true if this drift warrants manual inspection>
}

## EVALUATION RULES
1. Compare outputs for semantic meaning, not exact string match.
2. Flag any change in factual claims, safety behavior, or output structure as potentially significant.
3. A format change alone (e.g., bullet points vs. paragraphs) is low severity unless it breaks downstream parsing.
4. A refusal where the baseline answered, or vice versa, is always at least medium severity.
5. If either output contains hallucinations or unsupported claims, mark drift_severity as at least "medium" and set requires_human_review to true.
6. If you cannot determine whether drift occurred, set drift_detected to false but note uncertainty in drift_description.

## EXAMPLES
[EXAMPLES]

Return only the JSON object. Do not include markdown fences, explanations, or additional text.

Adaptation guidance: Replace [CONSTRAINTS] with domain-specific rules such as tone requirements, disallowed content, or compliance checks. The [EXAMPLES] placeholder should be populated with 2-3 few-shot examples showing clear drift vs. no-drift cases calibrated to your use case. If your golden dataset includes expected output text, add it as [EXPECTED_OUTPUT] and instruct the model to compare both versions against that reference. For high-risk domains, set requires_human_review to true by default for any drift_severity above "low" and route flagged examples to a review queue before accepting the prompt change.

The semantic drift detection prompt is designed to operate as a batch comparison engine within a CI/CD or release gate pipeline. You feed it pairs of outputs—one from the baseline prompt version and one from the candidate version—for each example in your golden dataset. The prompt returns a structured drift report containing per-example similarity scores, intent preservation flags, and clusters of affected output types. This harness should be invoked after your standard schema validation and exact-match checks pass, as semantic drift detection is more expensive and should only run on examples that survived structural validation.

To wire this into an application, construct a Python or Node.js evaluation script that iterates over your golden dataset, generates outputs from both prompt versions, and calls the LLM with the drift detection prompt for each pair. Implement a retry wrapper with exponential backoff (3 attempts minimum) to handle transient model failures. Parse the JSON response and validate it against a strict schema: each entry must contain example_id, similarity_score (float 0-1), intent_preserved (boolean), and drift_category (enum: none, minor, major, critical). Reject any response that fails schema validation and log the raw output for debugging. For high-stakes deployments, route critical drift examples to a human review queue before allowing promotion. Store all drift reports as versioned artifacts alongside your prompt commit hash for auditability.

Set your thresholds based on risk tolerance. A common starting point: block promotion if any example shows critical drift, require human review if more than 5% of examples show major drift, and allow minor drift with an automated changelog entry. Use a model with strong instruction-following and JSON output capabilities (such as gpt-4o or claude-3-5-sonnet) for the drift detection prompt itself, not a smaller or cheaper model that might introduce its own evaluation noise. Avoid running this prompt on every commit if your golden dataset is large; instead, trigger it on release candidates or PRs tagged for prompt review. Pair this with a cost-tracking log to prevent surprise bills from batch evaluation runs.