Confidence Threshold Routing Prompt

This prompt is for engineering teams building high-stakes classification or extraction pipelines where the cost of a wrong answer is high, but the cost of a full human review on every item is prohibitive. The job-to-be-done is to automatically branch the workflow: when a model's self-reported confidence in its primary output is above a calibrated threshold, the result is accepted and passed downstream. When confidence falls below the threshold, the system routes the input to a higher-effort path—a more detailed prompt variant, a stronger model, retrieval of additional evidence, or a human review queue. The ideal user is an AI engineer or backend developer integrating this logic into a production inference harness, not a business user manually copying a prompt into a chat interface.

You should use this prompt when your application has a measurable tolerance for error and you can define what 'uncertain' looks like for your specific task. This requires you to have already run a confidence calibration exercise: you must know the relationship between the model's raw confidence scores (logprobs, token probabilities, or verbalized scores) and actual accuracy on your data distribution. Without calibration data, a threshold is arbitrary and dangerous. The prompt template is designed to be wrapped in application code that parses the confidence score, compares it to a configured threshold, and executes the branch. It is not a standalone 'smart prompt' that magically decides when it is wrong.

Do not use this prompt for low-stakes tasks where the branching overhead is more expensive than the occasional error. Avoid it when you cannot extract a reliable confidence signal—some models, tasks, and output formats make confidence estimation noisy or miscalibrated. This prompt is also inappropriate when the high-effort fallback path is identical to the primary path; the whole point is a meaningful escalation in capability, evidence, or human attention. If you haven't instrumented your pipeline to log confidence scores, branch decisions, and outcomes, start there before deploying this prompt. Without observability, you cannot tune the threshold or detect silent failures.

This prompt template is the core instruction set for a confidence threshold routing system. It is designed to be assembled at runtime, where the model's primary task is to produce a structured output with a calibrated confidence score. The application layer then reads this score and routes the request to a high-effort variant, a human reviewer, or a fallback model if the score falls below the configured [CONFIDENCE_THRESHOLD]. The template uses square-bracket placeholders for all dynamic inputs, ensuring a clean separation between the static instruction logic and the variable data injected by your application harness.

textCopy
You are an expert [TASK_DOMAIN] classifier and extraction agent. Your primary objective is to analyze the provided [INPUT] and produce a structured output that strictly conforms to the [OUTPUT_SCHEMA].

## Instructions
1.  **Analyze the Input:** Carefully review the [INPUT] in the context of the provided [CONTEXT].
2.  **Generate Output:** Produce a JSON object that matches the [OUTPUT_SCHEMA] exactly. Do not include any keys that are not defined in the schema.
3.  **Calibrate Confidence:** For every major field or decision in your output, you must include a `confidence_score` between 0.0 and 1.0. A score of 1.0 represents absolute certainty, while 0.0 represents a complete guess. Base this score on the clarity of the evidence in the [INPUT] and [CONTEXT], not on your general knowledge.
4.  **Handle Ambiguity:** If the [INPUT] is ambiguous, contradictory, or lacks sufficient information to meet the [CONSTRAINTS], you must set the relevant `confidence_score` below the [CONFIDENCE_THRESHOLD] and populate the `ambiguity_notes` field in your output.
5.  **Apply Constraints:** You must adhere to all rules specified in [CONSTRAINTS]. If a constraint cannot be met, reflect this failure in the output's `confidence_score` and `error_flags`.

## Examples of Expected Behavior
[EXAMPLES]

## Available Tools
If you require external information to improve confidence, you may use the following tools. Only use a tool if it directly addresses a source of low confidence.
[TOOLS]

## Risk Context
The risk level for this task is [RISK_LEVEL]. If the risk level is 'high', you must be more conservative in your confidence calibration and explicitly flag any assumptions.

To adapt this template, start by defining your [OUTPUT_SCHEMA] as a strict JSON schema, ensuring it includes confidence_score, ambiguity_notes, and error_flags fields. The [CONFIDENCE_THRESHOLD] placeholder should be a float value (e.g., 0.85) injected by your application config. When providing [EXAMPLES], use few-shot demonstrations that explicitly show both high-confidence and low-confidence outputs, modeling the correct calibration behavior. The [TOOLS] placeholder should contain only the function definitions that are relevant to resolving ambiguity for this specific task, preventing the model from making unnecessary or distracting tool calls. Finally, ensure your application harness validates the output against the [OUTPUT_SCHEMA] before parsing the confidence score to make a routing decision.

The Confidence Threshold Routing Prompt is not a standalone artifact; it is a decision node in a broader inference pipeline. The core implementation pattern involves a two-stage architecture: a primary classification or extraction call that returns both a structured output and a calibrated confidence score, followed by a routing decision that either accepts the output, escalates to a high-effort variant, or triggers human review. This harness must be implemented in application code, not inside the prompt itself, because the routing logic requires deterministic threshold comparison, retry budget tracking, and audit logging that the model cannot reliably self-manage.

Start by defining a strict output contract for the primary prompt. The model must return a JSON object containing at minimum an output field with the task result and a confidence field as a float between 0.0 and 1.0. Implement a post-inference validator in your application layer that checks: (1) valid JSON parse, (2) confidence is present and within range, (3) output matches the expected schema for the task. If validation fails, do not route on confidence; instead, increment a retry counter and re-invoke the primary prompt with the validation error message appended as context. After two failed validation attempts, escalate directly to the high-effort variant or a human queue. For the routing decision itself, compare confidence against your predefined threshold (start with 0.85 and calibrate using eval data). Values at or above the threshold proceed; values below trigger the high-effort branch. Log every routing decision with the trace ID, input hash, confidence score, threshold used, branch taken, and latency for later calibration analysis.

The high-effort branch should be a separate prompt template with additional instructions: more detailed reasoning steps, explicit evidence citation requirements, counterfactual checks, or a request for the model to flag specific uncertainties. This variant will consume more tokens and add latency, so guard it with a per-session or per-window rate limit to prevent runaway costs. If the high-effort variant also returns confidence below a secondary, lower threshold (e.g., 0.7), route to a human review queue with the full context, both model outputs, and the confidence trail. Never silently accept low-confidence outputs in high-stakes pipelines. Instrument the entire harness with OpenTelemetry spans so you can trace every routing decision back to the input and model version that produced it. The most common production failure mode is threshold miscalibration: a model that consistently returns 0.95 confidence on wrong answers. Mitigate this by running a calibration eval weekly using a golden dataset with known ground truth, and adjust thresholds or switch to a model that produces better-calibrated scores.