Pairwise Semantic Similarity Regression Prompt for QA Pairs

This playbook is for evaluation infrastructure teams building automated regression gates for question-answering systems. When you modify a system prompt, upgrade a model, or change retrieval logic, you need to quantify whether the new outputs are semantically equivalent to the reference outputs. Simple string matching fails because a correct answer can be phrased many ways. This prompt produces a calibrated similarity score for each QA pair and a statistical drift detector that flags when the distribution of scores shifts beyond an acceptable variance. Use this when you need a repeatable, automated signal before promoting a prompt to production.

The ideal user is an AI platform engineer or QA lead who owns the release gate for a RAG system or structured QA bot. They have a golden dataset of question-reference answer pairs and need to run a batch comparison every time a candidate prompt is proposed. The prompt is designed to be wired into a CI/CD pipeline where it receives a batch of inputs, returns structured JSON with per-pair scores and a drift summary, and fails the gate if the mean similarity drops or the variance spikes beyond configured thresholds. It assumes the reference answers are factually correct and that semantic equivalence—not stylistic preference—is the primary quality dimension.

Do not use this prompt for open-ended creative generation, multi-turn conversations, or tasks where semantic equivalence is not the primary quality dimension. It is not a general-purpose quality judge; it is a focused regression detector for factual QA. Avoid using it when the reference answers are themselves unreliable, when the acceptable variance is undefined, or when the output format is unstructured prose rather than answer pairs. If your system requires human judgment for edge cases, use this prompt as a first-pass filter and escalate low-confidence or borderline pairs for manual review.

This template is the core instruction set for an LLM judge tasked with comparing outputs from a new prompt version against a golden set of reference answers. It is designed to be invoked programmatically for each QA pair in your regression dataset. The prompt instructs the model to produce a structured similarity score, a brief justification, and a flag for potential distribution shift, making it suitable for integration into an automated CI/CD evaluation harness.

codeCopy
You are a semantic regression analyst. Your task is to compare a new answer generated by an updated prompt against a reference answer from a golden QA dataset. You will produce a structured similarity assessment and a drift flag.

# INPUTS
- QUESTION: [QUESTION]
- REFERENCE ANSWER: [REFERENCE_ANSWER]
- NEW ANSWER: [NEW_ANSWER]
- ACCEPTABLE_VARIANCE: [ACCEPTABLE_VARIANCE]

# OUTPUT_SCHEMA
You must respond with a single JSON object conforming to this structure:
{
  "similarity_score": <float between 0.0 and 1.0>,
  "justification": "<one-sentence explanation of the score>",
  "drift_flag": <boolean>,
  "drift_reason": "<if drift_flag is true, explain the semantic shift; otherwise null>"
}

# INSTRUCTIONS
1. **Score Calculation**: Assign a `similarity_score` from 0.0 (completely different meaning) to 1.0 (semantically identical). Focus on factual accuracy and core intent, not superficial word choice.
2. **Drift Detection**: Compare the `similarity_score` to the `ACCEPTABLE_VARIANCE` threshold of [ACCEPTABLE_VARIANCE]. If the score is below this threshold, set `drift_flag` to `true`.
3. **Justification**: Provide a concise, evidence-based reason for the score, referencing specific details from the answers.
4. **Output Format**: Return only the valid JSON object. Do not include any other text, markdown fences, or commentary.

# CONSTRAINTS
- Do not hallucinate facts not present in either answer.
- Treat a non-answer or refusal by the new prompt as a score of 0.0 and a drift.
- If both answers are factually correct but differ in style or verbosity, the score should remain high (>= 0.85).

To adapt this template, replace the square-bracket placeholders with values from your dataset and test configuration. The [ACCEPTABLE_VARIANCE] is a critical control parameter; start with a value like 0.8 and tune it based on your application's tolerance for semantic drift. For high-stakes domains like healthcare or finance, set this threshold higher and combine this automated check with a human review step for any flagged drift. The output JSON can be ingested directly by your test harness to generate a distribution report and trigger a pass/fail gate.

This prompt is designed to be called programmatically within a regression testing harness, not used interactively. The harness iterates over a golden dataset of QA pairs, sending each [REFERENCE_OUTPUT] and [NEW_OUTPUT] to the model for scoring. The core integration pattern is a batch processing loop that collects similarity scores, stores them alongside metadata (prompt version, model, timestamp), and feeds the resulting distribution into a statistical drift detector. The prompt itself is stateless; all context must be provided per call.

For production implementation, wrap the prompt in a function that accepts a list of test cases and returns a structured RegressionRun object. Each call should include retry logic for transient API failures (exponential backoff, max 3 retries) and a timeout of 30 seconds per pair. Validate the model's response against the expected JSON schema before accepting the score. If the model returns a malformed JSON object or a score outside the 0.0–1.0 range, log the raw output and retry once with a stricter [CONSTRAINTS] block that emphasizes valid JSON only. For large datasets (>500 pairs), shard the work across parallel workers with a concurrency limit matching your API rate tier.

The statistical drift detector is the critical downstream component. After collecting all pairwise scores, compute the distribution's mean, median, standard deviation, and the Kolmogorov-Smirnov statistic against a stored baseline distribution from the previous prompt version. Set a release gate threshold: if the KS statistic exceeds 0.15 or the mean similarity drops below 0.85, block the promotion and flag the run for human review. Log every run as a structured artifact (JSON Lines format) with the prompt version hash, model identifier, timestamp, and full score distribution. This audit trail is essential for debugging regressions and justifying rollbacks. Avoid using this prompt for single-pair comparisons in isolation; its value comes from the aggregate distribution, not individual similarity scores.