Domain Confidence Scoring Rubric Prompt

This prompt is for routing systems that need more than a domain label. It produces a classification, a calibrated confidence score, and explicit reasoning so downstream logic can decide whether to auto-route, escalate for human review, or request clarification. The ideal user is an AI platform engineer or technical decision maker building a multi-tenant or multi-product system where the cost of misrouting is high—think legal intake, healthcare triage, or security incident dispatch—and where you need auditable decision traces for governance, debugging, or compliance reviews.

Use this prompt when you have a defined domain taxonomy and can tolerate a second inference call for confidence scoring after initial classification. It is not a replacement for simple keyword-based routing or a fast intent classifier with a hardcoded threshold. If your latency budget is under 200ms and you cannot afford a reasoning step, this prompt is the wrong tool. Similarly, if your routing logic is a single if/else on a label, you do not need a calibrated confidence score. This prompt assumes you have a downstream harness that can act on the score: auto-process above 0.9, escalate between 0.6 and 0.9, and ask the user for clarification below 0.6.

Before implementing, ensure you have a stable taxonomy, a set of representative test cases for each domain, and a plan for evaluating confidence calibration against actual routing outcomes. The biggest failure mode is treating the confidence score as an absolute truth rather than a signal that requires ongoing calibration. Start by running this prompt against a golden dataset of 200-500 labeled examples, compare predicted confidence to actual accuracy, and adjust your thresholds before routing live traffic. If you cannot commit to that calibration loop, use a simpler classification prompt and a static routing table instead.

This prompt template is designed to be pasted directly into your system instructions or user message. It instructs the model to classify an input into a domain from your taxonomy and produce a calibrated confidence score with structured reasoning. The output is a strict JSON schema that downstream routing middleware can parse to decide whether to auto-route, escalate for human review, or request clarification from the user. Replace every square-bracket placeholder with your specific values before use.

textCopy
You are a domain classification and confidence scoring system. Your task is to analyze the provided input and assign it to exactly one domain from the taxonomy below. You must also produce a confidence score between 0.0 and 1.0 and provide explicit, evidence-based reasoning for your score.

## TAXONOMY
[DOMAIN_TAXONOMY]

## INPUT
[USER_INPUT]

## INSTRUCTIONS
1. Identify the single best-matching domain from the taxonomy for the input.
2. Assign a confidence score from 0.0 (complete guess) to 1.0 (absolute certainty).
3. Provide a `reasoning` string that explains your confidence score. Reference specific signals from the input that support or weaken your classification. If the input is ambiguous, state which other domains were considered and why they were rejected.
4. If the input does not fit any domain, set `domain` to `null` and `confidence` to 0.0. Explain why in the reasoning.
5. If the input spans multiple domains, choose the primary domain and note the secondary domains in the reasoning.

## CONFIDENCE SCORING RUBRIC
- **0.9-1.0**: Unambiguous signals, domain-specific terminology, clear intent.
- **0.7-0.89**: Strong signals but minor ambiguity or missing detail.
- **0.5-0.69**: Plausible match but competing domains are possible.
- **0.3-0.49**: Weak signals, vague language, or high overlap with other domains.
- **0.0-0.29**: Very little signal, or input is out-of-scope.

## OUTPUT SCHEMA
You must respond with ONLY a valid JSON object. No markdown, no commentary.
{
  "domain": "string | null",
  "confidence": number,
  "reasoning": "string"
}

## CONSTRAINTS
- Do not invent domains not in the taxonomy.
- Do not return multiple domains in the `domain` field.
- The `reasoning` field must be specific to this input, not generic.
- If confidence is below [CONFIDENCE_THRESHOLD], recommend human review in the reasoning.

After pasting this template, replace [DOMAIN_TAXONOMY] with your actual taxonomy, formatted as a clear list or hierarchy (e.g., billing, technical_support, account_management, security_incident). Replace [USER_INPUT] with the text you want classified. Replace [CONFIDENCE_THRESHOLD] with your organization's threshold for auto-routing (e.g., 0.7). In production, wire the output JSON into a routing middleware that checks confidence against your threshold: if confidence is at or above the threshold, auto-route to the domain's handler; if below, escalate to a human review queue. Always validate the output JSON structure before acting on it, and log every classification decision with the input, output, and final routing action for audit and calibration analysis.

This prompt is designed to sit at a critical decision point in your routing pipeline: after domain classification but before dispatch. The harness must treat the prompt's output as a structured signal that downstream systems can act on deterministically. The core integration pattern is a classify-then-score sequence where the domain classifier produces a candidate label, and this rubric prompt independently evaluates that classification with a calibrated confidence score and reasoning chain. The harness should enforce that the confidence score is never used in isolation—it must always be paired with the reasoning field so that threshold decisions are auditable.

Validation and retry logic is essential before the score reaches any routing decision. Parse the JSON output and validate that confidence_score is a float between 0.0 and 1.0, reasoning is a non-empty string, and domain_label matches one of the expected taxonomy entries. If validation fails, retry once with the same input and an explicit error message injected into the prompt context. If the retry also fails, log the raw output and route the input to a human review queue with a confidence_override: null flag. For high-stakes domains like legal, finance, or healthcare, consider requiring confidence_score >= 0.85 for automated routing and flagging scores between 0.60 and 0.85 for expedited human review rather than full manual triage.

Model choice and latency budgeting directly impact this prompt's reliability. Use a model with strong instruction-following and JSON output discipline—GPT-4o, Claude 3.5 Sonnet, or equivalent—because weaker models tend to produce inconsistent score distributions or skip the reasoning field. Set temperature=0 or very low (0.1) to reduce score variance across identical inputs. If this prompt runs in a synchronous request path, budget 500-800ms for model inference and add a circuit breaker: if the model call exceeds 2 seconds, fall back to a default confidence_score: 0.5 with reasoning set to "timeout_fallback" and route to human review. For asynchronous batch classification, run this prompt in parallel across multiple inputs with a concurrency limit matching your model provider's rate limits.

Logging and calibration monitoring should be built into the harness from day one. Log every invocation with input_hash, classified_domain, confidence_score, reasoning, routing_decision, model_id, latency_ms, and validation_status. Periodically run a calibration eval: sample N recent classifications, have human reviewers assign true confidence labels, and compute Expected Calibration Error (ECE) to detect score drift. If ECE exceeds 0.1, investigate whether the prompt needs recalibration, the taxonomy has shifted, or the model version introduced scoring bias. Store these eval results alongside prompt version history so you can correlate calibration degradation with specific prompt or model changes.

Integration with the routing decision layer should be explicit and testable. After validation, pass the confidence score and reasoning to a routing function that implements your threshold matrix. For example: if confidence >= 0.9: auto_route(domain); elif confidence >= 0.7: route_with_soft_warning(domain); elif confidence >= 0.5: route_to_human_review(domain, reasoning); else: escalate_to_supervisor(domain, reasoning). Test this matrix with synthetic inputs at confidence boundaries (0.89, 0.90, 0.91) to verify no off-by-one routing errors. The harness should also detect when the same input hash produces different confidence scores across retries and flag those cases for prompt stability investigation.