Confidence Regression Test Prompt for RAG Eval Suites

This prompt is built for ML engineers and RAG system operators who need to detect confidence behavior drift before shipping a new prompt version. The core job-to-be-done is automated regression testing: given a fixed dataset of query-evidence pairs and two prompt versions, produce a structured comparison that flags whether the new version over-refuses, under-refuses, inflates confidence, or destabilizes threshold boundaries. Use this when you have a golden eval dataset and need a release gate for confidence-sensitive RAG systems in healthcare, legal, finance, or compliance contexts where wrong answers are worse than no answer.

Do not use this prompt for evaluating retrieval quality, answer correctness, or generation fluency. This prompt assumes you already have a working confidence extraction prompt and a labeled dataset. It compares confidence behavior, not answer quality. The ideal setup includes a fixed set of 100-500 query-evidence pairs with known expected confidence ranges, a baseline prompt version already running in production, and a candidate version you intend to ship. The output is a structured comparison report with drift metrics, threshold stability checks, and per-example deltas that you can use as a CI/CD gate.

Before running this prompt, ensure your eval dataset covers edge cases: borderline answerable queries, intentionally insufficient evidence, contradictory sources, and queries that should trigger refusal. Without these, the regression test will miss over-refusal and under-refusal patterns. After receiving the comparison output, review any examples where confidence shifted by more than 0.2 or where the abstention decision flipped. These are your release-blocking signals. If the new version passes all drift thresholds, you can proceed with deployment; if not, iterate on the prompt or adjust confidence thresholds before shipping.

This prompt template is the core instrument for your confidence regression test suite. It instructs the model to evaluate the same query-evidence pair using two different prompt versions and produce a structured comparison of their confidence behavior. You will wire this into your eval harness to detect drift in abstention thresholds, confidence score distributions, and tier assignments before shipping a prompt update. The template assumes you have a golden dataset of queries paired with retrieved evidence chunks, where each record has a known expected confidence outcome.

textCopy
You are a confidence regression test evaluator. Your task is to compare how two prompt versions handle the same query-evidence pair and report differences in confidence behavior.

## INPUT
Query: [QUERY]
Retrieved Evidence:
[EVIDENCE]

## PROMPT VERSION A (Baseline)
[PROMPT_VERSION_A]

## PROMPT VERSION B (Candidate)
[PROMPT_VERSION_B]

## OUTPUT SCHEMA
Return a JSON object with this exact structure:
{
  "version_a_output": {
    "answer": "string or null",
    "confidence_score": 0.0,
    "confidence_tier": "High|Medium|Low|Refuse",
    "abstained": true,
    "abstention_reason": "string or null"
  },
  "version_b_output": {
    "answer": "string or null",
    "confidence_score": 0.0,
    "confidence_tier": "High|Medium|Low|Refuse",
    "abstained": true,
    "abstention_reason": "string or null"
  },
  "comparison": {
    "confidence_drift": "none|minor|significant",
    "tier_changed": true,
    "abstention_flipped": true,
    "score_delta": 0.0,
    "key_differences": ["string describing each material difference"],
    "regression_detected": true,
    "regression_type": "over-refusal|under-refusal|score-collapse|tier-inflation|null"
  }
}

## CONSTRAINTS
1. Evaluate each prompt version independently against the same query and evidence.
2. Do not let Version B's output influence your assessment of Version A, or vice versa.
3. A regression is detected when Version B shows worse confidence behavior than Version A, defined as: answering when it should abstain (under-refusal), abstaining when it should answer (over-refusal), assigning High confidence to unsupported claims, or collapsing confidence scores toward extremes.
4. If either version produces an unparseable output, set the corresponding fields to null and flag the comparison with key_differences noting the parse failure.
5. Score delta is version_b confidence_score minus version_a confidence_score.
6. Confidence drift is "significant" when the score delta exceeds 0.2 or the tier changes, "minor" when the score delta is between 0.05 and 0.2 without tier change, and "none" otherwise.

## RISK_LEVEL
[HIGH_RISK_DOMAINS]

For queries in high-risk domains, flag any regression where Version B abstains less than Version A as a critical finding in key_differences, even if the score delta is small.

Adapt this template by replacing the placeholders with your actual test data and prompt versions. The [QUERY] and [EVIDENCE] fields come from your golden dataset. [PROMPT_VERSION_A] is your currently deployed prompt, and [PROMPT_VERSION_B] is the candidate you plan to ship. The [HIGH_RISK_DOMAINS] placeholder accepts a comma-separated list of domains—such as healthcare, legal, finance—where under-refusal regressions carry elevated risk. After running this prompt across your dataset, aggregate the comparison fields to produce drift reports, threshold stability charts, and a go/no-go signal for the candidate prompt. Do not ship Version B if regression_detected is true for any high-risk domain query without human review of each flagged case.

The confidence regression prompt is designed to run as a batch evaluation step, not a one-off manual check. You feed it a fixed dataset of query-evidence pairs plus two prompt versions, and it produces a structured comparison. The harness should treat this like any other CI step: versioned inputs, deterministic output schema, and a clear pass/fail gate. The primary integration points are your eval runner (e.g., Braintrust, LangSmith, a custom pytest suite), your model provider API, and your results store for trend analysis over time.

Start by loading your golden dataset—a JSONL file where each line contains query, retrieved_context (array of passage objects with text and source_id), and expected_behavior (one of should_answer, should_refuse, should_partial). For each record, call the confidence regression prompt with both the baseline and candidate prompt versions. Parse the JSON output and validate it against a strict schema: confidence_score must be a float 0-1, abstention_decision must be one of answer, refuse, partial, and evidence_sufficiency must be a float 0-1. If parsing fails, retry once with a repair prompt that includes the raw output and the schema. Log both attempts. After collecting all results, compute the regression metrics: mean confidence shift, abstention rate change, threshold stability (how many records cross your production confidence threshold between versions), and drift in evidence sufficiency scores. Gate deployment on a maximum allowed confidence shift (e.g., ±0.1 mean) and zero regressions on known refusal cases where the baseline correctly refused and the candidate answered.

Wire logging to capture per-record decisions, not just aggregate metrics. For every query where the candidate version changes the abstention decision or shifts confidence by more than 0.15, log the full record—query, evidence, both outputs, and the diff—to a review queue. These are your drift cases. Before gating a release, a human should spot-check the top 20 drift cases sorted by confidence shift magnitude. For high-stakes domains, require human sign-off on any candidate version that increases the answer rate on queries marked should_refuse in the golden dataset. Avoid the trap of tuning thresholds to make the regression report look clean; instead, treat threshold changes as a signal that your confidence calibration may have shifted and needs separate investigation.