Routing Confidence Audit Prompt

This prompt is a retrospective audit tool for governance, quality assurance, and reliability engineering teams responsible for the health of production classification systems. Its primary job is to review a batch of routing decisions—each paired with its confidence score, input context, and routing metadata—and produce a structured audit trail that flags potential misroutes, confidence anomalies, and systematic failure patterns. The ideal user is an engineer or analyst who needs to answer questions like: 'Are we silently misrouting requests?' 'Is our confidence threshold calibrated correctly?' or 'Are there entire categories of input where our classifier has drifted?' This is not a prompt for real-time routing; it is an offline diagnostic instrument for compliance review, pre-release QA, and ongoing system health monitoring.

To use this prompt effectively, you must provide a batch of routing records in a structured format. Each record should include the original input, the routed destination (e.g., a queue, intent label, or model name), the confidence score, and any relevant metadata such as the classifier version or processing timestamp. The prompt's value scales with the quality of this input data. A batch of 50 records with sparse metadata will yield a shallow audit; a batch of 500 records with rich context will reveal calibration drifts and taxonomy gaps. The output is a structured audit report containing a list of flagged decisions, each with a severity level, a reason code (e.g., HIGH_CONFIDENCE_MISROUTE, CONFIDENCE_CALIBRATION_ANOMALY), and a human-readable explanation. This report is designed to be consumed by a human reviewer or ingested into a monitoring dashboard.

There are clear situations where this prompt should not be used. Do not use it for real-time routing decisions; it is too slow and verbose for a request path. Do not use it as a substitute for a proper evaluation framework with a held-out ground-truth dataset; it can flag suspicious patterns but cannot measure precision and recall. Do not use it on a single routing decision; its strength lies in pattern detection across a batch. Finally, do not treat its output as a final verdict. The prompt is a triage tool that directs human attention to the highest-risk decisions. Every flag it raises should be reviewed by a person who understands the system's taxonomy and the business impact of a misroute. The next step after running this audit is to take the flagged records and either correct them, use them to improve your classification prompts, or adjust your confidence thresholds.

This prompt template is designed to be copied directly into your AI harness for batch auditing of routing decisions. It expects a structured record of each routing event—including the original input, the assigned route, the confidence score, and any relevant context—and returns a structured audit finding for each record. The square-bracket placeholders represent the dynamic data and policy controls you must inject before sending the prompt to the model. Replace each placeholder with your actual routing records, taxonomy definitions, confidence thresholds, and audit criteria.

textCopy
You are a routing quality auditor. Your task is to review a batch of routing decisions and flag potential misroutes, confidence anomalies, and systematic failure patterns.

## ROUTING TAXONOMY
[TAXONOMY]

## CONFIDENCE THRESHOLDS
- High confidence: score >= [HIGH_THRESHOLD]
- Medium confidence: score between [MEDIUM_THRESHOLD] and [HIGH_THRESHOLD]
- Low confidence: score < [MEDIUM_THRESHOLD]

## AUDIT CRITERIA
[CRITERIA]

## ROUTING RECORDS TO AUDIT
[ROUTING_RECORDS]

## OUTPUT SCHEMA
For each routing record, produce a JSON object with these fields:
- record_id: string (the original record identifier)
- audit_result: "pass" | "flag" | "uncertain"
- finding_type: "misroute" | "confidence_anomaly" | "systematic_pattern" | "threshold_violation" | "none"
- severity: "critical" | "high" | "medium" | "low" | "info"
- explanation: string (specific evidence for the finding, citing the input, assigned route, and confidence score)
- recommended_action: "reclassify" | "human_review" | "adjust_threshold" | "investigate_pattern" | "no_action"
- confidence_score_quality: "calibrated" | "overconfident" | "underconfident" | "unclear"

## CONSTRAINTS
- Do not reclassify inputs; only audit the existing routing decision.
- Flag a record only when there is specific evidence of a problem, not just low confidence.
- If multiple records show the same failure pattern, note the pattern in each affected record and add a cross-reference to the first instance.
- For any record flagged as "critical" severity, include a specific recommendation for immediate action.
- If the routing taxonomy is ambiguous for a given input, note the ambiguity in the explanation.

## EXAMPLES
[EXAMPLES]

Return a JSON array of audit findings, one object per routing record. Do not include commentary outside the JSON.

To adapt this template, start by defining your [TAXONOMY] as a structured list of valid routes with descriptions and example inputs for each. This gives the model a clear reference for judging whether an input was routed correctly. Your [CRITERIA] should enumerate the specific conditions that constitute a misroute, a confidence anomaly, or a systematic pattern—for example, 'flag when a high-confidence route contradicts the input's explicit intent' or 'flag when three or more records in the same category show confidence scores dropping below the medium threshold within a single batch.' The [ROUTING_RECORDS] placeholder should receive a JSON array of objects, each containing at minimum the original input text, the assigned route label, the confidence score, and any metadata your audit criteria require. Provide at least three [EXAMPLES] that demonstrate correct pass, flag, and uncertain audit results so the model understands the expected judgment calibration. After generating audit findings, always run them through a validation step that checks every record_id in the input appears exactly once in the output, that all severity and finding_type values match the allowed enums, and that critical-severity findings include actionable recommendations. For high-stakes routing systems—such as those handling financial transactions, healthcare triage, or safety-critical decisions—route all flagged findings to a human reviewer before taking automated corrective action.

The Routing Confidence Audit Prompt is designed to be a batch processing step within a governance pipeline, not a real-time decision point. It expects a structured log of prior routing decisions—each containing the original input, the assigned route, the confidence score, and any available metadata—and produces a structured audit report. To integrate this into an application, you should first aggregate routing decisions from your production middleware into a JSON Lines file or a database view that can be paginated into batches. The prompt is stateless, so each invocation should receive a complete, self-contained batch of records. The batch size should be tuned to stay within the model's context window while leaving room for the output report; 50–100 records per batch is a practical starting point for most models.

The implementation harness should wrap the model call with pre-validation, post-validation, and logging layers. Before calling the model, validate that each input record contains the required fields: input_text, assigned_route, confidence_score, and routing_timestamp. Reject any malformed records and log them separately. After receiving the model's output, parse the JSON report and validate it against a strict schema: the flagged_records array must contain objects with record_id, flag_type (one of misroute, confidence_anomaly, systematic_pattern), severity, and evidence. If parsing or schema validation fails, implement a retry loop with a maximum of 2 attempts, appending the validation errors to the next prompt as [PREVIOUS_ERRORS]. If retries are exhausted, escalate the entire batch to a human review queue with the raw model output and validation failure details attached. For high-stakes audit workflows, consider routing all severity: critical findings to a human reviewer for confirmation before the audit report is finalized, even when the model output is technically valid.

Logging is essential for auditability. Each invocation should produce an immutable log entry containing the batch ID, the full prompt sent, the raw model response, the parsed and validated output, any validation errors, retry counts, and the final disposition (success, partial_success, escalated). Store these logs in a system that supports querying by batch ID and timestamp. For model choice, use a model with strong reasoning capabilities and reliable JSON output, such as Claude 3.5 Sonnet or GPT-4o, and set temperature=0 to maximize consistency across audit runs. Do not use this prompt with smaller or faster models that may hallucinate flag types or produce malformed JSON, as the cost of a missed systematic misroute pattern far outweighs the inference savings. The next step after a successful audit run is to feed the systematic_patterns array into your prompt improvement or model retraining pipeline, closing the loop from detection to remediation.