Confidence Threshold Calibration Prompt Template

This prompt is for ML engineers who need to move from a raw model confidence score to a calibrated, production-ready decision threshold. The core job-to-be-done is replacing an arbitrary threshold (like 0.7 or 0.9) with a data-driven recommendation that balances precision and recall against your specific business requirements. You should use this prompt when you have a holdout set of scored predictions with ground-truth labels and need an executable starting point for an automated quality gate. The ideal user is someone who can supply a CSV or JSON dataset of [score, label] pairs and understands the operational cost of false positives versus false negatives in their system.

Do not use this prompt as a replacement for a full MLOps calibration pipeline. It will not perform isotonic regression, Platt scaling, or temperature scaling on your model outputs. It is designed to analyze the scores you already have and recommend an operating point. Avoid using it when your holdout set is too small to be statistically meaningful, when your production distribution has drifted significantly from your evaluation data, or when the cost of a single misclassification is so high that every decision requires human review. In regulated or high-risk domains, treat the prompt's output as an initial analysis that must be reviewed by a qualified engineer before it gates any real traffic.

After running this prompt, you will receive a recommended threshold, a precision-recall trade-off table, and calibration diagnostics. Your next step is to version this output alongside your model artifacts, write a unit test that asserts the threshold produces acceptable metrics on a golden dataset, and wire the threshold into your deployment harness with monitoring for drift. The prompt gives you a starting point; your engineering discipline makes it production-grade.

This template is designed for ML engineers who need to move beyond a single hardcoded threshold and produce a data-driven calibration report. The prompt expects a set of model outputs, their ground-truth correctness labels, and the raw confidence scores the model assigned. It returns threshold recommendations at multiple operating points, a precision-recall curve summary, and explicit calibration checks against a provided holdout set. Use this when you are integrating a new model into a production quality gate, tuning an existing gate after a data distribution shift, or documenting threshold decisions for an audit.

textCopy
You are a production ML quality engineer calibrating confidence thresholds for an AI output gating system.

## INPUT DATA
- Model Outputs with Scores: [OUTPUT_SCORE_DATASET]
- Ground Truth Labels: [GROUND_TRUTH_LABELS]
- Holdout Set for Validation: [HOLDOUT_DATASET]
- Current Production Threshold (if any): [CURRENT_THRESHOLD]

## CONSTRAINTS
- Business Requirements: [BUSINESS_CONSTRAINTS]
  (Example: "Maximum 5% false negative rate for critical outputs, prefer higher precision for automated actions")
- Risk Level: [RISK_LEVEL]
  (Options: LOW, MEDIUM, HIGH, CRITICAL)
- Downstream Action: [DOWNSTREAM_ACTION]
  (Example: "auto-publish", "route-to-review", "block-and-alert")

## OUTPUT SCHEMA
Return a valid JSON object with this exact structure:
{
  "calibration_report": {
    "generated_at": "ISO 8601 timestamp",
    "dataset_summary": {
      "total_samples": int,
      "positive_class_count": int,
      "negative_class_count": int,
      "score_distribution": {
        "min": float,
        "max": float,
        "mean": float,
        "median": float,
        "std": float
      }
    },
    "threshold_recommendations": [
      {
        "threshold": float,
        "operating_point_name": string,
        "precision": float,
        "recall": float,
        "f1_score": float,
        "false_positive_rate": float,
        "false_negative_rate": float,
        "expected_pass_rate": float,
        "recommended_for": string
      }
    ],
    "primary_recommendation": {
      "threshold": float,
      "rationale": string,
      "trade_off_analysis": string
    },
    "holdout_validation": {
      "holdout_samples": int,
      "primary_threshold_precision": float,
      "primary_threshold_recall": float,
      "calibration_error": float,
      "overfit_warning": boolean
    },
    "drift_monitoring_hooks": {
      "score_distribution_metrics": [string],
      "alert_thresholds": {
        "pass_rate_drop_below": float,
        "mean_score_shift_above": float
      },
      "recommended_recalibration_cadence": string
    },
    "limitations_and_risks": [string]
  }
}

## INSTRUCTIONS
1. Analyze the score distribution and separate the data into calibration and validation splits if no holdout is provided.
2. Compute precision, recall, F1, and false positive/negative rates at 10-15 threshold points spanning the score range.
3. Select 3-5 operating points that represent distinct trade-offs (high-precision, balanced, high-recall).
4. Recommend a primary threshold based on the business constraints and risk level.
5. Validate the primary threshold against the holdout set and report calibration error.
6. Define drift monitoring hooks: which metrics to track and what shifts should trigger recalibration.
7. If RISK_LEVEL is HIGH or CRITICAL, include a specific warning about the consequences of threshold drift and recommend a human-review fallback for scores within 0.05 of the threshold.
8. Do not invent data. If the dataset is too small for reliable calibration (fewer than 200 samples), include a limitation note and recommend collecting more data before deploying.

Adaptation notes: Replace each square-bracket placeholder with your actual data and constraints. The [OUTPUT_SCORE_DATASET] should be a JSON array of objects with at least id, score, and raw_output fields. The [GROUND_TRUTH_LABELS] should map each id to a boolean or binary label. If you do not have a separate holdout set, the prompt will split the provided data automatically, but explicitly providing [HOLDOUT_DATASET] produces more trustworthy validation. For high-risk or critical deployments, always run this calibration on a dataset that reflects your current production distribution—stale calibration data is the most common cause of threshold failure. After pasting, test the prompt with a small known dataset before running it on your full production data.

This prompt is not a one-shot script; it is a calibration step inside a larger ML pipeline. The harness should treat the LLM as an analyst proposing thresholds, not as the final decision authority. The prompt expects a [CALIBRATION_DATA] payload containing precision-recall curves, confidence distributions, and business constraints. The harness must validate that the model's output—a set of recommended thresholds with trade-off analysis—is structurally sound before any threshold is applied to a production quality gate. A common failure mode is accepting a well-formatted but mathematically inconsistent recommendation, so the harness must include numeric validation checks.

The implementation should follow a validate → retry → escalate loop. First, parse the model's JSON output and validate that all recommended thresholds are within [0,1], that precision and recall values are monotonic with respect to threshold changes, and that the trade-off analysis references the provided [CONSTRAINTS] (e.g., minimum recall, maximum false-positive rate). If validation fails, retry with the original prompt plus the validation error message appended to [CONTEXT]. Limit retries to 2 attempts. If retries are exhausted, log the failure, flag the calibration run for human review, and fall back to the previous known-good threshold set. Use a model with strong JSON mode and numeric reasoning (e.g., gpt-4o or claude-3.5-sonnet) and set temperature=0 to minimize variance in threshold recommendations.

After a valid recommendation is produced, the harness must run a holdout evaluation before promoting the threshold to production. Split [CALIBRATION_DATA] into a training set (used in the prompt) and a holdout set (used for post-hoc verification). Apply the recommended threshold to the holdout set and confirm that the achieved precision and recall fall within an acceptable tolerance (e.g., ±5%) of the model's claims. If the holdout check fails, log the discrepancy as a calibration drift event and escalate. Additionally, log every calibration run's inputs, outputs, validation results, and holdout metrics to a calibration audit table for drift monitoring over time. This audit trail is essential for debugging quality regressions and for compliance reviews.

Finally, integrate the accepted threshold into your production gating system with a circuit breaker. Monitor the rate of outputs flagged as low-confidence. If the flag rate deviates by more than 20% from the calibration baseline, trigger an alert and consider re-running the calibration prompt with fresh data. Do not silently apply thresholds from a stale calibration run. The harness should treat the prompt output as a point-in-time recommendation, not a permanent configuration.