Session Risk Confidence Score Calibration Prompt

This prompt is for ML engineers and safety platform builders who need to calibrate the confidence scores of a session-level risk classifier. The core job-to-be-done is converting raw model logits or uncalibrated probabilities into reliable, interpretable confidence scores that can gate downstream actions like tool denial, human handoff, or session termination. You should use this prompt when you have a classifier that produces per-turn or per-session risk scores, a human-labeled ground truth dataset of session risk, and a need to understand the reliability curve of your classifier before deploying it to production. The ideal user has access to model outputs, ground truth labels, and a calibration library like scikit-learn or ludwig.

Do not use this prompt if you are still building the initial risk classifier, if you lack human-labeled session ground truth, or if you are looking for a single-turn confidence calibration. This prompt assumes you already have a working risk score signal and need to measure and improve its calibration. It is also not a replacement for a full ML evaluation pipeline; it is a focused calibration diagnostic that produces reliability curves, expected calibration error (ECE), and uncertainty bands. The output is meant to inform threshold selection and deployment gating, not to serve as the final production scoring endpoint without additional validation.

Before running this prompt, ensure you have prepared a calibration dataset containing at minimum: the classifier's raw risk scores, the human-labeled ground truth risk labels, and session identifiers for traceability. The prompt expects these inputs as structured data and will produce calibration diagnostics. After calibration, you should validate the calibrated scores on a held-out test set, compare against single-turn confidence baselines, and document the reliability curve for your model card. If the calibrated scores still show high ECE or narrow uncertainty bands that don't cover true outcomes, you may need to revisit feature engineering or consider isotonic regression as an alternative calibration method.

This prompt template is designed to take a session-level risk classifier's raw outputs and produce calibrated confidence scores, reliability curves, and diagnostic metrics. It expects the classifier's predictions alongside human-labeled ground truth for the same sessions. The output includes calibration error metrics, uncertainty bands, and a comparison against single-turn confidence baselines so you can quantify how much session context improves reliability. Use this template when you have a batch of scored sessions and need to validate whether the classifier's confidence numbers mean what they claim before deploying to production gating or escalation workflows.

textCopy
You are a calibration analyst for a session-level safety risk classifier. Your task is to compare the classifier's predicted risk scores against human-labeled ground truth and produce calibrated confidence scores with diagnostic metrics.

## INPUTS
- Classifier predictions: [CLASSIFIER_PREDICTIONS]
  Format: JSON array of objects with fields: session_id, predicted_risk_score (0.0-1.0), predicted_confidence (0.0-1.0), turn_count, risk_category
- Human-labeled ground truth: [GROUND_TRUTH_LABELS]
  Format: JSON array of objects with fields: session_id, true_risk_label (binary 0/1), true_risk_severity (0.0-1.0), labeler_count, agreement_score (0.0-1.0)
- Single-turn baseline scores: [SINGLE_TURN_BASELINE]
  Format: JSON array of objects with fields: session_id, max_single_turn_risk_score, max_single_turn_confidence

## OUTPUT SCHEMA
Return a JSON object with this exact structure:
{
  "calibration_summary": {
    "expected_calibration_error": number,
    "maximum_calibration_error": number,
    "brier_score": number,
    "num_sessions": integer,
    "num_bins": integer
  },
  "reliability_curve": [
    {
      "bin_lower": number,
      "bin_upper": number,
      "mean_predicted": number,
      "mean_actual": number,
      "bin_count": integer
    }
  ],
  "uncertainty_bands": {
    "confidence_intervals": [
      {
        "confidence_level": number,
        "lower_bound": number,
        "upper_bound": number
      }
    ],
    "prediction_interval_width": number
  },
  "session_level_comparison": {
    "session_accuracy": number,
    "session_precision": number,
    "session_recall": number,
    "session_f1": number
  },
  "single_turn_comparison": {
    "single_turn_accuracy": number,
    "single_turn_precision": number,
    "single_turn_recall": number,
    "single_turn_f1": number,
    "session_vs_single_turn_delta": {
      "accuracy_delta": number,
      "f1_delta": number
    }
  },
  "calibration_diagnostics": [
    {
      "diagnostic": string,
      "value": number,
      "interpretation": string,
      "severity": "low" | "medium" | "high" | "critical"
    }
  ],
  "flagged_sessions": [
    {
      "session_id": string,
      "predicted_risk_score": number,
      "predicted_confidence": number,
      "true_risk_label": integer,
      "miscalibration_type": "overconfident" | "underconfident" | "correct_direction_wrong_magnitude",
      "miscalibration_magnitude": number
    }
  ],
  "recommendations": [string]
}

## CONSTRAINTS
- Use equal-frequency binning with [NUM_BINS] bins for the reliability curve.
- Flag sessions where |predicted_risk_score - true_risk_severity| > [CALIBRATION_THRESHOLD] as miscalibrated.
- Only include sessions where agreement_score >= [MIN_AGREEMENT_THRESHOLD] in calibration calculations.
- Weight sessions by labeler_count when computing aggregate metrics.
- If fewer than [MIN_SESSIONS] sessions are provided, return an error with explanation instead of partial results.
- Report expected calibration error (ECE) using L1 norm across bins.
- For uncertainty bands, compute at [CONFIDENCE_LEVELS] levels using bootstrapping with [BOOTSTRAP_SAMPLES] resamples.
- Compare session-level metrics against single-turn baseline and compute deltas.
- Flag diagnostics where ECE > 0.1 as "high" severity and ECE > 0.2 as "critical".

## EVALUATION CRITERIA
- Calibration error must be computed correctly per bin.
- Flagged sessions must include specific miscalibration type and magnitude.
- Recommendations must reference specific diagnostic findings, not generic advice.
- Single-turn comparison must use the same metric definitions for valid delta computation.
- Output must validate against the schema above.

## EXAMPLES
[EXAMPLES]

## TOOLS
[TOOLS]

Adaptation notes: Replace each square-bracket placeholder with your actual data and configuration. [CLASSIFIER_PREDICTIONS], [GROUND_TRUTH_LABELS], and [SINGLE_TURN_BASELINE] should be populated from your evaluation pipeline before sending this prompt. [NUM_BINS] typically defaults to 10 for reliability curves. [CALIBRATION_THRESHOLD] should be set based on your risk tolerance—0.15 is a common starting point. [MIN_AGREEMENT_THRESHOLD] filters out sessions where human labelers disagreed, preventing garbage-in garbage-out calibration; 0.7 is a reasonable floor. [MIN_SESSIONS] prevents calibration on insufficient data; require at least 100 sessions for meaningful ECE computation. [CONFIDENCE_LEVELS] can be an array like [0.8, 0.9, 0.95] and [BOOTSTRAP_SAMPLES] should be at least 1000 for stable intervals. If your classifier outputs per-turn scores rather than session aggregates, preprocess them into session-level predictions before calling this prompt. For production use, wrap this prompt in a validation harness that checks the output JSON against the schema and retries with error context if validation fails.

What to do next: After running calibration, feed the flagged_sessions back into your labeling queue for review—these are the sessions where your classifier is most confidently wrong. Use the session_vs_single_turn_delta to justify session-level modeling investment to stakeholders. If ECE exceeds 0.1, do not deploy the classifier to automated gating; instead, apply Platt scaling or isotonic regression to the raw scores and recalibrate. Store the reliability_curve and calibration_diagnostics alongside your model card for audit evidence. Avoid the common mistake of treating calibration as a one-time check—recalibrate whenever your user population, attack patterns, or base model change.

This prompt is designed for offline calibration, not for real-time inference gating. It expects a batch of scored sessions—each containing turn-level risk features, a raw model score, and a human-labeled ground truth—and returns calibrated confidence scores, reliability curves, and calibration diagnostics. The primary integration point is an MLOps evaluation pipeline that runs after a new risk classifier version is trained or when drift is suspected. Do not call this prompt synchronously in a request path; its latency and token cost are appropriate for periodic batch jobs.

Wire the prompt into a Python evaluation harness that loads a labeled session dataset, formats each record into the [SESSION_BATCH] placeholder, and calls the model with response_format set to the [OUTPUT_SCHEMA] JSON Schema. Validate the returned JSON immediately: confirm that calibrated_sessions contains an entry for every input session ID, that reliability_curve bins are non-overlapping and cover the 0–1 range, and that calibration_error fields are present and numeric. On validation failure, retry once with the error message appended to the prompt as a correction hint. Log every calibration run with a unique run_id, the model version used, the dataset split name, and the full calibration output for auditability.

The harness must compute two comparison baselines: single-turn confidence calibration (using only the first turn's score per session) and uncalibrated raw session scores. Store these alongside the calibrated output so you can track whether session-level calibration meaningfully improves expected calibration error (ECE) and reliability diagram sharpness. If the calibrated ECE is not lower than the single-turn baseline, flag the run for review—this often indicates that the session risk features are not adding signal beyond the first turn, or that the human labels contain systematic noise.

For high-risk deployment contexts, add a human review gate before accepting calibration parameters into production. The review packet should include the reliability curve plot, ECE before and after calibration, the 10 sessions with the largest calibration shift, and a drift comparison against the previous calibration run. Only promote the new calibration if a reviewer confirms that the shifts are justified and not artifacts of label error or model regression. This gate prevents a silent degradation where a poorly calibrated confidence score causes over-refusal or under-refusal in the downstream session risk gating system.