Safety Model Confidence Evaluation Rubric Prompt

This prompt is for ML engineers and safety platform builders who have already run inference with a safety classifier and now need to evaluate whether the model's confidence scores are well-calibrated. The job-to-be-done is a rigorous, automated calibration audit: you have a held-out test set containing ground-truth labels and predicted confidence scores, and you need a structured evaluation report that includes Expected Calibration Error (ECE), a reliability diagram data table, and a diagnostic summary. This is not a prompt for generating safety classifications—it is a meta-evaluation prompt that judges the quality of scores your classifier already produced.

Use this prompt when you are preparing to ship a threshold-based gating system and must verify that a confidence score of 0.8 actually means the classifier is correct 80% of the time. It is appropriate when you have a representative test set with at least several hundred examples spanning clear violations, edge cases, and benign requests. The prompt assumes your classifier outputs a confidence score between 0 and 1 for each input, and that you have ground-truth binary labels for comparison. You should also use this prompt after retraining or fine-tuning a safety model, when migrating between model versions, or when you observe unexpected refusal patterns in production and need to diagnose whether miscalibration is the root cause.

Do not use this prompt when you lack ground-truth labels, when your test set is too small for reliable binning, or when you need real-time scoring rather than offline evaluation. This prompt is also inappropriate for evaluating the classifier's accuracy or recall directly—it focuses specifically on calibration quality. If you need to generate safety classifications from raw inputs, use the Safety Classification Confidence Scoring Prompt Template instead. If you need to configure routing thresholds based on calibration results, pair this evaluation with the Risk Threshold Gating Decision Prompt. Before running this evaluation, ensure your test set covers the full range of confidence scores and includes sufficient examples in high-risk bins where miscalibration is most dangerous.

This prompt template is designed to be dropped into an evaluation harness that feeds a batch of predictions from your safety classifier. It instructs the model to act as a calibration auditor, comparing predicted confidence scores against ground-truth accuracy. The output is a structured report containing Expected Calibration Error (ECE) and the data needed to plot a reliability diagram, which are the primary artifacts for diagnosing overconfidence or underconfidence in production safety systems.

textCopy
You are an ML evaluation auditor. Your task is to evaluate the calibration of a safety classifier's confidence scores.

You will receive a set of predictions in JSON format. Each prediction includes:
- "input_text": the text that was classified.
- "predicted_label": the class predicted by the safety classifier.
- "confidence_score": the model's confidence in its prediction (a float between 0 and 1).
- "true_label": the ground-truth label from the held-out test set.

[INPUT_DATA]

Analyze the provided predictions and produce a structured calibration report. Follow these steps:
1. Bucket the predictions by confidence score into deciles (0.0-0.1, 0.1-0.2, ..., 0.9-1.0).
2. For each bucket, calculate:
   - The number of samples.
   - The average confidence score.
   - The observed accuracy (fraction of samples where predicted_label equals true_label).
3. Calculate the Expected Calibration Error (ECE) as the weighted average of the absolute difference between average confidence and accuracy across all buckets.
4. Identify the bucket with the largest calibration gap.
5. Provide a diagnostic summary: state whether the model is generally well-calibrated, overconfident, or underconfident, and note any specific confidence ranges that are unreliable.

[OUTPUT_SCHEMA]
Return your analysis as a single JSON object with the following structure:
{
  "ece": <float>,
  "reliability_diagram_data": [
    {
      "confidence_bucket": "<string, e.g., '0.0-0.1'>",
      "sample_count": <int>,
      "average_confidence": <float>,
      "accuracy": <float>,
      "gap": <float>
    }
  ],
  "worst_calibrated_bucket": "<string>",
  "diagnostic_summary": "<string>"
}

[CONSTRAINTS]
- If a bucket has zero samples, omit it from the reliability_diagram_data array.
- The diagnostic_summary must be a single paragraph under 150 words.
- Do not include any text outside the JSON object.

To adapt this prompt, replace the [INPUT_DATA] placeholder with your serialized JSON array of predictions. The [OUTPUT_SCHEMA] and [CONSTRAINTS] placeholders are defined inline for clarity but can be externalized if you maintain a shared schema library. For high-stakes safety evaluations, always run this prompt on a statistically significant sample (at least 1,000 examples) and log the output for auditability. If the ECE exceeds a pre-defined threshold (e.g., 0.05), your next step should be to investigate the worst-calibrated bucket and consider recalibrating the classifier using Platt scaling or isotonic regression before re-deploying.

This prompt is designed to be executed as a batch evaluation job, not a real-time API call. It should be triggered whenever a new safety classifier model version is deployed, when the held-out test set is updated, or on a fixed cadence (e.g., weekly) to detect calibration drift. The primary integration point is your model evaluation pipeline, such as a CI/CD step in your MLOps platform or a scheduled notebook execution. The prompt expects a pre-assembled payload containing predicted confidence scores and ground-truth accuracy labels from a static test set. Do not run this on live production traffic; it is a meta-evaluation tool for the classifier itself.

To wire this into an application, build a pre-processing script that queries your model registry for the latest safety classifier and runs inference on the held-out calibration set. The script must construct the [TEST_SET] placeholder as a JSON array of objects, each containing predicted_confidence (float 0-1), actual_correct (boolean), and an optional case_id for traceability. The [CONSTRAINTS] placeholder should be populated with your organization's specific calibration tolerance, such as a maximum Expected Calibration Error (ECE) of 0.05. After the LLM call, a post-processing validator must parse the JSON output and enforce the [OUTPUT_SCHEMA]. If the model returns malformed JSON, implement a retry with a stricter schema instruction. Log the raw prompt, the full response, and the parsed evaluation to your experiment tracker (e.g., MLflow, Weights & Biases) for auditability.

The primary failure mode is the LLM hallucinating calibration statistics instead of computing them from the provided data. To mitigate this, your post-processing script should independently calculate the Expected Calibration Error (ECE) and bin-wise accuracy from the raw [TEST_SET] data using a deterministic library like sklearn.calibration. Compare the LLM's reported ECE against the programmatic calculation. If the absolute difference exceeds a small epsilon (e.g., 0.01), flag the evaluation as unreliable and escalate for human review. For high-risk safety systems, always require a human to sign off on the final evaluation report before promoting a new classifier model. The reliability diagram data generated by the prompt should be plotted in your monitoring dashboard to visually confirm calibration quality.