Trace-Based Confidence Score Regression Prompt

Use this prompt when you are an AI ops engineer or observability lead who needs to compare the confidence score distributions of two prompt versions using real production traces. The primary job is to detect calibration drift—shifts in how the model assigns confidence relative to actual outcomes—before a regression silently degrades downstream decision thresholds. This prompt is designed for teams that already have production trace data with ground-truth outcome labels (e.g., task success, user correction, or human review verdicts) and need a structured report to decide whether to roll back, recalibrate, or promote a new prompt version. It is not a general-purpose prompt evaluator; it assumes you have trace-level confidence scores and outcome labels ready to analyze.

You should not use this prompt when you lack outcome-labeled traces, when you are comparing fewer than 100 traces per version (statistical noise will dominate), or when your confidence scores come from different model families with incompatible calibration baselines. This prompt also does not replace a full regression test suite—it focuses narrowly on confidence calibration drift. If you need to detect semantic drift, instruction adherence regressions, or format compliance issues, pair this with the sibling prompts for semantic drift detection or format compliance regression. For high-stakes domains such as healthcare triage, loan eligibility, or safety-critical decisions, always route the output through a human reviewer and cross-reference with outcome-based calibration metrics before acting on the report.

Before running this prompt, ensure you have: (1) a baseline trace dataset from the current production prompt version, (2) a comparison trace dataset from the candidate prompt version, (3) confidence scores and binary outcome labels for each trace, and (4) the decision thresholds your application uses to gate actions. The prompt expects these inputs structured as arrays of objects with confidence, outcome, and optional category fields. If your traces span multiple intent categories or risk levels, segment the analysis by category to avoid masking category-specific drift. After receiving the output, validate the calibration metrics against your own outcome data, log the report as an artifact in your release gate, and do not promote the candidate version if the overconfidence flag is raised in a high-risk category.

The following prompt template is designed to be copied directly into your AI harness or evaluation pipeline. It instructs the model to act as a calibration analyst, comparing two sets of production traces—typically a baseline (current) version and a candidate (new) version—to detect and explain shifts in confidence score behavior. The prompt is structured to produce a machine-readable report that flags overconfidence, underconfidence, and decision-threshold impacts, which can then be parsed by downstream monitoring or release-gate systems.

codeCopy
You are a calibration analyst reviewing production traces from two versions of an AI system.

Your task is to compare the confidence score distributions between the [BASELINE_VERSION] traces and the [CANDIDATE_VERSION] traces to detect calibration drift.

## INPUT DATA

### Baseline Traces ([BASELINE_VERSION])
[BASELINE_TRACES]

### Candidate Traces ([CANDIDATE_VERSION])
[CANDIDATE_TRACES]

## CONFIDENCE SCORE DEFINITION
[CONFIDENCE_SCORE_DEFINITION]

## DECISION THRESHOLDS
[DECISION_THRESHOLDS]

## OUTPUT SCHEMA
Produce a JSON object with the following structure:
{
  "calibration_drift_detected": boolean,
  "drift_summary": {
    "baseline_mean_confidence": number,
    "candidate_mean_confidence": number,
    "mean_shift": number,
    "baseline_std_confidence": number,
    "candidate_std_confidence": number,
    "distribution_shape_change": string ("narrower" | "wider" | "shifted_right" | "shifted_left" | "no_significant_change"),
    "statistical_test_applied": string,
    "p_value_or_effect_size": number
  },
  "overconfidence_flags": [
    {
      "confidence_bucket": string,
      "baseline_accuracy_in_bucket": number,
      "candidate_accuracy_in_bucket": number,
      "accuracy_drop": number,
      "severity": string ("low" | "medium" | "high"),
      "affected_thresholds": [string],
      "example_trace_ids": [string]
    }
  ],
  "underconfidence_flags": [
    {
      "confidence_bucket": string,
      "baseline_accuracy_in_bucket": number,
      "candidate_accuracy_in_bucket": number,
      "accuracy_gain": number,
      "severity": string ("low" | "medium" | "high"),
      "affected_thresholds": [string],
      "example_trace_ids": [string]
    }
  ],
  "decision_threshold_impact": [
    {
      "threshold": string,
      "baseline_pass_rate": number,
      "candidate_pass_rate": number,
      "pass_rate_change": number,
      "risk": string ("more_conservative" | "more_aggressive" | "no_change"),
      "recommended_action": string
    }
  ],
  "overall_assessment": string,
  "recommended_investigations": [string]
}

## CONSTRAINTS
[CONSTRAINTS]

## ANALYSIS INSTRUCTIONS
1. Group confidence scores into buckets: [0.0-0.5), [0.5-0.7), [0.7-0.8), [0.8-0.9), [0.9-0.95), [0.95-1.0].
2. For each bucket, calculate the actual accuracy (proportion of correct outcomes) for both versions.
3. Flag any bucket where the gap between confidence and accuracy increased by more than [OVERCONFIDENCE_THRESHOLD] as overconfidence.
4. Flag any bucket where accuracy exceeds confidence by more than [UNDERCONFIDENCE_THRESHOLD] as underconfidence.
5. For each decision threshold in [DECISION_THRESHOLDS], calculate the proportion of traces that would pass in each version.
6. If [RISK_LEVEL] is "high", include a statement that this analysis requires human review before any release decision.
7. Only reference trace IDs that exist in the provided input data.
8. If fewer than [MIN_SAMPLE_SIZE] traces exist in a bucket, mark severity as "low" and note the small sample size.

To adapt this template for your environment, replace the square-bracket placeholders with concrete values. The [BASELINE_TRACES] and [CANDIDATE_TRACES] fields should contain serialized trace objects that include at minimum a trace ID, the model's confidence score, and the ground-truth outcome. The [CONFIDENCE_SCORE_DEFINITION] should explain how confidence is derived in your system—for example, "the maximum softmax probability of the output token" or "the model's self-reported certainty on a 0-1 scale." The [DECISION_THRESHOLDS] field should list the confidence cutoffs your application uses to route, escalate, or auto-approve outputs. The [CONSTRAINTS] field is where you inject domain-specific rules, such as "do not flag overconfidence if the absolute accuracy remains above 95%" or "ignore buckets related to the 'chitchat' intent category." For high-stakes domains, set [RISK_LEVEL] to "high" to enforce the human-review guardrail in the output. Always validate the output JSON against the schema before ingesting it into your regression dashboard or release gate.

This prompt is designed to run as a batch analysis step, not a real-time user-facing call. It expects two structured datasets: a baseline confidence score distribution from production traces under the current prompt version, and a candidate distribution from traces under the new prompt version. The harness should fetch these from your observability store (e.g., a trace database, log analytics platform, or feature store) and format them as JSON arrays of per-prediction confidence scores, optionally segmented by intent, risk tier, or decision threshold. Run this prompt once per comparison window—typically during a canary deployment, a staged rollout, or a pre-release QA gate—not on every individual trace.

Wire the prompt into a scripted evaluation pipeline. Before calling the model, validate that both input arrays are non-empty, contain numeric values in [0,1], and have sufficient sample sizes per segment to produce meaningful comparisons (at least 50 samples per segment is a reasonable floor). After receiving the model output, parse the JSON report and run automated checks: confirm the calibration_drift object contains expected keys (ece_delta, overconfidence_shift, underconfidence_shift), verify that affected_thresholds lists actual decision thresholds from your system, and check that severity is one of the allowed enum values. If parsing fails, retry once with a stricter output schema instruction appended to the prompt. Log the raw prompt, the model response, and the parsed report to your evaluation store for auditability.

Integrate the parsed report into your release gate logic. Define quantitative thresholds that trigger a block or a warning—for example, an Expected Calibration Error (ECE) increase greater than 0.05, an overconfidence flag on a high-risk decision threshold, or a statistically significant shift detected by the calibration test. When a regression is flagged, the harness should surface the recommended_action field to the release manager and attach the full report to the deployment review. Avoid treating the prompt's severity classification as the sole gate decision; always combine it with outcome-based metrics (e.g., actual error rates, user correction rates) from your production monitoring. For high-stakes domains, require human review of any flagged regression before promoting the new prompt version.