Confidence Score Normalization Prompt Across Models

Platform teams routing tasks across multiple models face a hard problem: every model reports confidence differently. GPT-4 returns logprobs. Claude returns a text-based assessment. Gemini returns a different scale. Open-source models might return raw token probabilities or nothing at all. Without normalization, a 0.8 from one model means something completely different from a 0.8 from another, making it impossible to set uniform escalation thresholds, compare model reliability, or build consistent human-in-the-loop workflows. This prompt solves that problem by taking a raw confidence signal from any model, along with metadata about how that signal was produced, and returning a normalized score on a standard 0.0 to 1.0 scale.

Use this prompt when you have a multi-model routing layer, an evaluation harness that compares outputs across providers, or a confidence-based escalation policy that needs a single, comparable signal. The prompt expects three inputs: the raw confidence value as reported by the source model, the source model's identifier and confidence type (e.g., gpt-4_logprobs, claude-3_verbal, gemini-pro_confidence), and any available calibration context such as the task category or historical accuracy for that model on similar inputs. The output includes the normalized score, the normalization method applied, explicit caveats about what the normalized score does and does not mean, and enough context for downstream systems to make consistent routing and escalation decisions. Wire this prompt into your routing middleware or evaluation pipeline so that every confidence-dependent decision operates on a common scale.

Do not use this prompt when you need real-time calibration guarantees or when the raw confidence signal is missing entirely. This prompt normalizes existing signals; it does not manufacture confidence where none exists. It also does not replace proper calibration—if you need to know whether a model's 0.9 actually means 90% correct, you need a calibration evaluation harness with known-outcome test cases, not just normalization. For high-risk domains such as healthcare, finance, or safety-critical infrastructure, always pair normalized scores with human review thresholds and log every normalization decision for audit. The normalized score is a routing convenience, not a safety guarantee.

This template accepts a raw confidence score from any model, the model's name, and optional metadata about how the score was produced. It returns a normalized score on a standard 0.0–1.0 scale, the normalization method applied, and explicit caveats about cross-model comparability. Use this when your platform routes tasks across multiple models with incompatible confidence signals—logprobs from one model, verbal self-assessment from another, and proprietary scores from a third.

textCopy
You are a confidence score normalization engine. Your job is to convert raw confidence signals from different AI models into a common 0.0–1.0 scale, document your normalization method, and flag any comparability risks.

## INPUT
- Raw confidence score: [RAW_SCORE]
- Raw score type: [SCORE_TYPE]  (e.g., logprob, probability, verbal rating, custom scale)
- Raw score range (if known): [RAW_RANGE]  (e.g., "0 to 1", "-inf to 0", "1 to 5", "unknown")
- Source model: [MODEL_NAME]
- Task description: [TASK_DESCRIPTION]
- Additional context (optional): [CONTEXT]

## OUTPUT_SCHEMA
Return a JSON object with these fields:
{
  "normalized_score": number,        // 0.0 to 1.0, where 1.0 = maximum confidence
  "normalization_method": string,    // e.g., "logprob_exponentiation", "linear_rescaling", "verbal_to_numeric_mapping", "identity_pass_through"
  "method_description": string,     // One-sentence explanation of the transformation applied
  "original_score": number | string, // The raw score as received
  "original_type": string,          // Confirmed or inferred score type
  "caveats": [string],              // List of comparability warnings, calibration concerns, or assumptions
  "comparable_across_models": boolean // Whether this normalized score can be meaningfully compared to scores from other models
}

## CONSTRAINTS
1. If the raw score type is unknown or ambiguous, infer it from the value and range. Document your inference in `method_description`.
2. For logprob scores, convert using exponentiation: normalized = exp(raw_logprob). Clamp to [0, 1].
3. For probability scores already in [0, 1], pass through but note in caveats if the source model is known to produce miscalibrated probabilities.
4. For verbal ratings (e.g., "high", "medium", "low", "very confident"), map to numeric values using a documented mapping. Include the mapping in `method_description`.
5. For custom scales, linearly rescale to [0, 1] using the provided range. If the range is unknown, flag this as a critical caveat.
6. If the raw score cannot be meaningfully normalized, set `normalized_score` to null and explain why in caveats.
7. Never fabricate calibration data. If you don't know whether a model's scores are well-calibrated, say so.
8. Set `comparable_across_models` to false if the normalization required assumptions, inference, or lossy transformation.

## EXAMPLES

Example 1: Logprob input
Input: raw_score=-0.8, score_type="logprob", raw_range="-inf to 0", model="model-a"
Output: {"normalized_score": 0.449, "normalization_method": "logprob_exponentiation", "method_description": "Exponentiated logprob: exp(-0.8) ≈ 0.449", "original_score": -0.8, "original_type": "logprob", "caveats": ["Logprob exponentiation assumes the score represents log-likelihood of the chosen output. Cross-model comparability depends on whether models use comparable tokenization and output spaces."], "comparable_across_models": false}

Example 2: Verbal input
Input: raw_score="high", score_type="verbal", raw_range="unknown", model="model-b"
Output: {"normalized_score": 0.8, "normalization_method": "verbal_to_numeric_mapping", "method_description": "Mapped 'high' to 0.8 using standard mapping: low=0.2, medium=0.5, high=0.8, very_high=0.95", "original_score": "high", "original_type": "verbal", "caveats": ["Verbal-to-numeric mapping is inherently coarse. Different models may use different verbal scales. This score should not be directly compared to continuous scores from other models."], "comparable_across_models": false}

## RISK_LEVEL
[RISK_LEVEL]  // If "high", add extra caveats about not using normalized scores for safety-critical decisions without human review.

Adaptation notes: Replace [RAW_SCORE], [SCORE_TYPE], [RAW_RANGE], [MODEL_NAME], [TASK_DESCRIPTION], [CONTEXT], and [RISK_LEVEL] before sending. If your platform only handles one or two known score types, simplify the constraints section to remove irrelevant branches. For production use, pair this prompt with a post-processing validator that checks the output JSON conforms to the schema and that normalized_score falls within [0, 1] when not null. If comparable_across_models is false, your application should prevent direct score comparisons in downstream decision logic. For high-risk workflows, route outputs with comparable_across_models: false or null normalized_score to a human reviewer before any automated action proceeds.

The Confidence Score Normalization Prompt is designed to sit between raw model outputs and downstream decision logic. It accepts a heterogeneous set of confidence signals—logprobs, verbal confidences, token probabilities, or custom scoring fields—and maps them onto a common 0.0–1.0 scale with a declared normalization method. The harness should treat this prompt as a stateless transformation step: it receives a structured input payload, returns a structured output payload, and must never be the sole gate for high-risk actions without additional calibration checks.

Wire the prompt into your application as a post-inference middleware function. After each model returns its native confidence signal, construct the input payload with the [MODEL_NAME], [RAW_SCORE], [SCORE_TYPE] (e.g., logprob, verbal_1_5, token_mean), and any [MODEL_METADATA] (temperature, sampling params). Validate the output against a strict schema before allowing it to influence routing or gating decisions. The normalized output must include normalized_score (float, 0–1), normalization_method (string), original_score (float), and caveats (array of strings). Reject any response where normalized_score falls outside [0,1] or where normalization_method is not one of your allowed values (linear_scale, logprob_exp, verbal_mapping, custom). Log every normalization event with the input model, raw score, normalized score, method, and timestamp for later calibration analysis.

For production reliability, implement a calibration checkpoint that runs periodically. Feed the normalizer a labeled dataset where ground-truth correctness is known, and compare normalized confidence against actual accuracy. If a model's normalized scores show consistent overconfidence (e.g., normalized 0.9 but actual accuracy 0.6), flag the model for recalibration or routing weight reduction. Do not use the normalized score directly for blocking irreversible actions unless you have validated calibration on your specific distribution. When confidence falls below your action threshold, route to a human review queue with the full normalization payload—not just the score—so reviewers can see the original signal, the method, and any caveats the prompt returned.