Confidence Threshold Calibration Prompt

This prompt is for MLOps engineers and AI platform teams who need to move from a single, gut-feel confidence threshold to a set of per-category thresholds that are justified by data. Use it when you have a batch of classification outputs with model-assigned confidence scores and corresponding ground-truth labels. The prompt instructs the model to act as a threshold calibration analyst: it computes precision, recall, and F1 scores at multiple candidate thresholds for each intent category, then recommends the threshold that best balances your stated business objective (e.g., minimize false positives, maximize recall, or hit a specific precision target). This is not a prompt for live classification. It is an offline tuning prompt that produces a threshold configuration you then deploy into your routing middleware.

The ideal user has a labeled evaluation dataset where each record contains the input text, the model's predicted intent, the model's raw confidence score, and the ground-truth label. You must also articulate a clear business objective per category—for example, 'maximize recall for the billing_dispute intent because missed disputes create regulatory risk' or 'require 95% precision for the account_closure intent because false positives trigger irreversible actions.' Without labeled data, this prompt cannot produce useful thresholds. If you lack ground-truth labels, use the Classification Confidence Scoring Prompt Template instead. If you need to explain individual confidence scores, use the Confidence Score Explanation Prompt.

Before running this prompt, ensure your evaluation dataset covers edge cases, overlapping intents, and production-representative distributions. A threshold tuned on clean, balanced data will fail silently when deployed against noisy production traffic. After generating thresholds, validate them against a held-out test set and monitor for drift using the Confidence Drift Monitoring Prompt. Never deploy per-category thresholds without a fallback path for inputs that fall below all thresholds—pair this output with the Low-Confidence Routing Fallback Prompt to define what happens when no category threshold is met.

This prompt template is designed to be the core instruction set for your threshold calibration pipeline. It takes a batch of classification outputs, their associated confidence scores, and the ground-truth labels, then produces a data-driven recommendation for the optimal confidence threshold per intent. The goal is to balance precision and recall according to your specified business objective, such as minimizing false positives for high-risk categories or maximizing recall for broad-coverage intents.

textCopy
You are a classification threshold calibration engine. Your task is to analyze a set of classification results against ground-truth labels and recommend an optimal confidence threshold for each intent category.

## INPUT DATA
You will receive the following structured data:
- [CLASSIFICATION_OUTPUTS]: A JSON array of objects, each with `id`, `predicted_intent`, `confidence_score` (0.0 to 1.0), and `all_scores` (a map of intent -> score).
- [GROUND_TRUTH]: A JSON array of objects, each with `id` and `true_intent`.
- [INTENT_CATEGORIES]: A JSON array of all possible intent strings.
- [OPTIMIZATION_OBJECTIVE]: A string describing the business goal, e.g., "Maximize F1-score", "Maximize recall with minimum 0.90 precision", or "Minimize false positives for [HIGH_RISK_INTENT]".

## CALIBRATION INSTRUCTIONS
For each intent in [INTENT_CATEGORIES], perform the following analysis:
1.  **Calculate Metrics:** For thresholds from 0.0 to 1.0 in 0.05 increments, calculate the true positives, false positives, true negatives, false negatives, precision, recall, and F1-score.
2.  **Apply Objective:** Identify the threshold that best satisfies the [OPTIMIZATION_OBJECTIVE]. If the objective specifies a constraint (e.g., min precision), the threshold must meet it.
3.  **Stability Check:** For the recommended threshold, calculate the same metrics at +/- 0.05 to report a stability range. Flag the threshold as "UNSTABLE" if any metric changes by more than 10%.
4.  **Handle Edge Cases:** If an intent has fewer than [MIN_SAMPLES] examples in the [GROUND_TRUTH], do not recommend a threshold. Instead, set the recommendation to "INSUFFICIENT_DATA" and suggest a fallback threshold of [DEFAULT_FALLBACK_THRESHOLD].

## OUTPUT SCHEMA
Return your analysis as a single valid JSON object conforming to this structure:
{
  "threshold_recommendations": [
    {
      "intent": "string",
      "recommended_threshold": "number | 'INSUFFICIENT_DATA'",
      "metrics_at_threshold": {
        "precision": "number",
        "recall": "number",
        "f1_score": "number",
        "true_positive_rate": "number",
        "false_positive_rate": "number"
      },
      "stability": {
        "status": "STABLE | UNSTABLE",
        "metrics_at_plus_005": { "precision": "number", "recall": "number", "f1_score": "number" },
        "metrics_at_minus_005": { "precision": "number", "recall": "number", "f1_score": "number" }
      },
      "rationale": "A brief, data-driven explanation for the recommendation."
    }
  ],
  "global_summary": "A summary of the calibration run, including the overall best F1-score and any intents with insufficient data."
}

## CONSTRAINTS
- Do not invent data. Base all analysis strictly on the provided [CLASSIFICATION_OUTPUTS] and [GROUND_TRUTH].
- If the [OPTIMIZATION_OBJECTIVE] is impossible to satisfy for a given intent, state this clearly in the rationale and recommend the closest possible threshold.
- The output must be valid, parseable JSON with no additional text or markdown formatting outside the code block.

To adapt this template, replace the square-bracket placeholders with your actual data and parameters. The [CLASSIFICATION_OUTPUTS] and [GROUND_TRUTH] should be populated from your evaluation dataset, typically a held-out test set that wasn't used in training. The [OPTIMIZATION_OBJECTIVE] is a critical business lever; a high-risk compliance intent might use "Minimize false positives with recall >= 0.95," while a general FAQ intent might use "Maximize F1-score." The [MIN_SAMPLES] and [DEFAULT_FALLBACK_THRESHOLD] parameters prevent the model from making brittle recommendations on sparse data. Before deploying any recommended threshold, always run the stability check and review the flagged "UNSTABLE" intents, as these are the first to break under data drift.

The Confidence Threshold Calibration Prompt is designed to operate as a batch analysis step within an MLOps pipeline, not as a real-time inference endpoint. It consumes a structured dataset of classification outputs paired with ground-truth labels and produces threshold recommendations per intent category. The prompt expects a JSON array of classification records—each containing the predicted intent, the model's raw confidence score, and the verified ground-truth label—alongside configuration parameters such as the target precision floor, minimum recall, and the cost-weighting ratio for false positives versus false negatives. Because this prompt performs numerical optimization reasoning, it should run against a model with strong quantitative reasoning capabilities and a large context window to accommodate substantial evaluation datasets.

To integrate this prompt into a production calibration pipeline, wrap it in a harness that performs three stages: input validation, prompt execution with retry, and output validation with stability checks. The input validation stage must verify that the provided dataset contains the required fields (predicted_intent, confidence_score, ground_truth_label), that confidence scores are numeric and within [0,1], and that the dataset covers all intent categories present in the taxonomy. Reject malformed inputs before model invocation to avoid wasted inference cost. The execution stage should call the prompt with a structured JSON output instruction and implement a retry loop (maximum 3 attempts) that catches JSON parse failures, schema mismatches, or missing required fields in the response. Each retry should append the specific validation error to the prompt context so the model can self-correct. The output validation stage must confirm that every recommended threshold is a float between 0 and 1, that each intent category in the input taxonomy has a corresponding threshold recommendation, and that the returned precision and recall estimates at each threshold are internally consistent with the provided data. Log every calibration run—including the input dataset hash, model version, prompt version, raw output, validation errors, retry count, and final thresholds—to an audit table for traceability and rollback capability.

Beyond single-run validation, the harness must include a threshold simulation step that applies the recommended thresholds to a held-out calibration subset and computes actual precision, recall, and F1 scores per intent category. Compare these simulated metrics against the prompt's self-reported estimates; discrepancies greater than 5 percentage points should trigger a warning and block automated threshold deployment. Additionally, implement a stability check that runs the calibration prompt across multiple time windows or data slices and flags any intent category where the recommended threshold shifts by more than 0.1 between slices. Large threshold drift indicates either data distribution shift or prompt sensitivity that requires investigation before the new thresholds are promoted to production routing rules. The final output of the harness should be a versioned threshold configuration artifact—a JSON file mapping each intent to its calibrated threshold, precision target, and minimum recall—that can be consumed directly by your classification router's configuration system. Never auto-deploy thresholds without human review when the stability check fails, when any intent category falls below the minimum recall floor, or when the calibration dataset contains fewer than 50 ground-truth examples for a given intent.