Risk Threshold Mapping Prompt

This prompt is designed for platform engineers and AI operations teams who need to normalize risk scores, confidence levels, and severity classifications across multiple model providers. The core job-to-be-done is producing a set of threshold mapping rules that translate a risk score from one model's native scale (e.g., a 0.0–1.0 confidence score from an OpenAI classifier) to another's (e.g., a categorical LOW/MEDIUM/HIGH from a Claude model) without losing the intended business logic. The ideal user is someone responsible for a multi-model routing or evaluation pipeline where a single risk threshold must trigger the same action—escalation, human review, or automatic rejection—regardless of which model generated the score.

Use this prompt when you have at least two models with documented confidence or risk scales and you need a deterministic mapping table, not a heuristic guess. You must provide the native score ranges, the target unified schema, and the business rules that define what constitutes an acceptable false-positive or false-negative rate. Do not use this prompt for general sentiment analysis, vague 'riskiness' scores without a defined scale, or when the underlying models have not been calibrated on a shared evaluation dataset. The prompt is a calibration tool, not a substitute for running actual cross-model evaluations.

Before running this prompt, ensure you have a validation dataset with ground-truth labels that both models have scored. The output mapping rules must be tested against this dataset to verify threshold equivalence. If the mapping produces a significant shift in escalation rates, you likely have a model capability gap, not a mapping problem. In high-stakes domains like healthcare or finance, the output of this prompt is a starting point for a human-reviewed risk policy, not a final automated decision layer. Pair this prompt with the 'Cross-Model Safety Policy Equivalence Check' playbook for a complete validation workflow.

This prompt template is designed to normalize risk scores, confidence levels, and severity classifications across multiple model providers. It produces threshold mapping rules that ensure consistent risk-based decision-making regardless of which model generates the initial score. The template uses square-bracket placeholders that must be replaced with your specific risk categories, target models, and acceptable tolerance ranges before use.

codeCopy
You are a risk calibration engineer responsible for normalizing risk scores across different AI models. Your task is to produce a threshold mapping specification that ensures consistent risk-based decisions regardless of which model generated the original score.

## INPUT DATA

### Risk Categories and Severity Levels
[RISK_CATEGORIES]

### Source Model Outputs
For each risk category, you will receive confidence scores from multiple models. Each model uses a different scale and distribution.

[SOURCE_MODEL_OUTPUTS]

### Target Decision Thresholds
The system must make binary decisions (proceed/escalate) based on normalized risk levels.

[TARGET_THRESHOLDS]

### Acceptable Tolerance Ranges
Maximum acceptable deviation between models for equivalent risk levels.

[TOLERANCE_RANGES]

## OUTPUT SCHEMA

Produce a JSON object with the following structure:

[OUTPUT_SCHEMA]

## CONSTRAINTS

[CONSTRAINTS]

## CALIBRATION RULES

1. For each risk category, analyze the score distribution from each source model.
2. Identify the effective threshold on each model's native scale that corresponds to the target decision boundary.
3. Document any non-linear relationships between confidence scores and actual risk.
4. Flag categories where models disagree beyond the acceptable tolerance range.
5. Provide per-model threshold values that produce equivalent decision outcomes.

## VALIDATION REQUIREMENTS

After producing the mapping, validate it by:
- Checking that equivalent-risk inputs produce the same decision across models
- Identifying edge cases where thresholds diverge
- Documenting confidence intervals for each mapped threshold

## OUTPUT FORMAT

Return a JSON object containing:
- `threshold_mappings`: Array of per-category, per-model threshold values
- `equivalence_groups`: Groups of thresholds that produce equivalent decisions
- `divergence_flags`: Categories where models cannot be calibrated within tolerance
- `validation_results`: Summary of equivalence checks performed
- `recommendations`: Suggested actions for high-divergence categories

To adapt this template, replace each square-bracket placeholder with concrete values. [RISK_CATEGORIES] should contain your specific risk taxonomy with severity definitions. [SOURCE_MODEL_OUTPUTS] requires actual score samples from each model you're calibrating. [TARGET_THRESHOLDS] defines your decision boundaries. [TOLERANCE_RANGES] sets acceptable calibration error. [OUTPUT_SCHEMA] should specify the exact JSON fields your downstream system expects. [CONSTRAINTS] should include any domain-specific rules, regulatory requirements, or operational limits that affect threshold placement. Always validate the output against a held-out test set before deploying threshold mappings to production decision systems.

This prompt is designed to run as part of a threshold calibration pipeline, not as a one-off chat interaction. The typical harness receives a set of provider-specific risk score definitions, severity taxonomies, and target equivalence rules, then calls the prompt once per provider pair or once for a full multi-provider mapping table. Because the output directly controls downstream routing, escalation, and human-in-the-loop decisions, the harness must validate every mapping row before it reaches production configuration. Treat the prompt output as a candidate artifact that requires automated validation, human review for high-risk thresholds, and version-controlled storage alongside the system instructions it governs.

Wire the prompt into an application by first assembling the input context: extract each provider's documented confidence scale, risk score range, severity labels, and any existing threshold definitions from your model gateway or policy registry. Pass these as structured fields inside [PROVIDER_A_SPEC] and [PROVIDER_B_SPEC], using a consistent JSON schema so the prompt receives normalized inputs regardless of provider. After the model returns the mapping table, run a threshold equivalence validator that checks: (1) every severity level in Provider A maps to exactly one severity level in Provider B, (2) numeric threshold boundaries are monotonic and non-overlapping, (3) the mapping preserves the intended escalation behavior defined in [EQUIVALENCE_RULES], and (4) no threshold gap leaves a risk score unclassified. If validation fails, retry the prompt once with the validator's error messages appended to [CONSTRAINTS] as explicit correction instructions. If the second attempt also fails, flag the provider pair for manual calibration and log the full input-output trace for review.

For model choice, prefer a model with strong instruction-following and structured output reliability—GPT-4o, Claude 3.5 Sonnet, or Gemini 2.0 Flash with JSON mode enabled all work well. Set temperature to 0 or near-zero to maximize deterministic threshold placement. Store every generated mapping in a version-controlled configuration store (e.g., a Git-backed policy registry or a database with audit timestamps) alongside the validator results, the prompt version hash, and the identity of any human reviewer who approved the mapping. Before deploying a new threshold mapping to production, run a shadow evaluation: route a sample of recent decisions through both the old and new thresholds and compare escalation rates, refusal rates, and human-review triggers. A significant shift in any of these metrics should block the rollout until the mapping is re-examined. Avoid wiring threshold mappings directly into live routing without this shadow step—silent threshold drift is the most common failure mode in multi-provider risk systems.