Model Routing Decision Audit Prompt

This prompt is a diagnostic tool for AI platform engineers and SREs who need to audit a single production trace to verify that the model routing decision was correct. The job-to-be-done is forensic: you have a trace where the system chose a specific model, and you need to determine whether that choice aligned with your routing policy. The prompt analyzes the user input, classification scores, routing rules, and selected model to determine if the cheapest or fastest appropriate model was chosen according to policy. Use this when you suspect routing logic is misconfigured, when cost anomalies appear in your inference spend, or when you need to validate that fallback and escalation paths are working as designed.

This is a trace review prompt, not a real-time routing prompt. It expects a complete trace payload containing the original user input, any classification or intent scores, the routing rules that were active at request time, the model that was selected, and the available model catalog with cost and capability metadata. The prompt does not make routing decisions—it audits decisions that have already been made. The ideal user is someone who understands the routing architecture and can supply the trace data in a structured format. If you are looking for a prompt that performs live model routing at inference time, use a prompt from the Model Routing and Fallback Prompts pillar instead.

Do not use this prompt when the trace data is incomplete or when routing rules were applied by an external system that cannot be reconstructed. The audit is only as reliable as the trace evidence provided. For high-stakes routing decisions—such as those involving regulated data, safety-critical workflows, or financial transactions—always pair this prompt's output with human review and cross-reference against your routing configuration as deployed. The prompt identifies likely misconfigurations and policy violations, but it cannot access live routing state or confirm that the trace accurately reflects production behavior. After running this audit, feed the findings into your routing rule refinement process and consider adding a continuous monitoring check that compares actual routing against expected policy across multiple traces.

This template is designed to be copied directly into your observability tool, notebook, or automated audit harness. It accepts a structured trace payload and a routing policy document, then produces a deterministic audit report. The prompt forces the model to compare the actual routing decision against the expected policy, not to generate a plausible-sounding justification for whatever happened. Use it when you need to answer one question: 'Did the router pick the right model for this request?'

textCopy
You are an AI platform auditor. Your task is to review a single production trace and determine whether the model routing decision was correct according to the provided routing policy.

## INPUT

### Trace Data
```json
[TRACE_DATA]

Routing Policy

textCopy
[ROUTING_POLICY]

INSTRUCTIONS

1. Extract the following from the trace:

   • The user's original input.
   • Any classification scores, intents, or features used for routing.
   • The model that was actually selected.
   • The routing rule that was applied (if recorded).

2. Apply the routing policy to the extracted features. Determine which model should have been selected according to the policy.

3. Compare the actual model against the expected model. If they differ, identify the specific policy rule that was violated or misapplied.

4. Classify the routing decision as one of:

   • CORRECT: The actual model matches the expected model.
   • SUBOPTIMAL: A cheaper or faster model could have been used without violating policy constraints.
   • OVERRIDE: The actual model is more expensive or slower than the expected model, and no policy exception is recorded.
   • POLICY_VIOLATION: The actual model violates an explicit policy constraint (e.g., data residency, capability requirement).
   • INDETERMINATE: The trace lacks sufficient information to apply the policy.

5. If the decision is not CORRECT, estimate the cost and latency impact of the deviation. Use the pricing and latency baselines provided in the policy. If baselines are absent, note that impact cannot be calculated.

OUTPUT SCHEMA

Return a JSON object with this exact structure:

jsonCopy
{
  "trace_id": "string",
  "timestamp": "string or null",
  "extracted_features": {
    "input_summary": "string",
    "classification_scores": {},
    "routing_signals_used": []
  },
  "actual_model": "string",
  "expected_model": "string",
  "applied_rule": "string or null",
  "decision": "CORRECT | SUBOPTIMAL | OVERRIDE | POLICY_VIOLATION | INDETERMINATE",
  "deviation_detail": "string or null",
  "estimated_impact": {
    "cost_delta_usd": "number or null",
    "latency_delta_ms": "number or null",
    "impact_notes": "string"
  },
  "audit_notes": "string"
}

CONSTRAINTS

• Do not invent missing trace data. If a signal is absent, mark it as null and classify as INDETERMINATE.
• Do not justify the actual decision. Report what the policy requires.
• If the policy is ambiguous, describe the ambiguity in audit_notes and classify as INDETERMINATE.
• Never recommend a model that violates data residency, compliance, or capability constraints stated in the policy.
• If the trace contains multiple routing steps, evaluate the final model selection only.

EXAMPLES

[EXAMPLES]

To adapt this template, replace [TRACE_DATA] with the full JSON trace object from your observability system. Replace [ROUTING_POLICY] with your organization's routing rules in plain text, including model tiers, cost baselines, latency targets, and any hard constraints like data residency. The [EXAMPLES] placeholder should contain 2-3 few-shot examples showing correct audit outputs for known routing scenarios. Before deploying this prompt in an automated pipeline, validate its output against a golden set of 20-30 traces where you know the correct routing decision. Run the audit on traces from each decision category to confirm the model isn't biased toward classifying everything as CORRECT.

The Model Routing Decision Audit Prompt is designed to be called programmatically as part of an automated trace review pipeline, not as a one-off manual check. In a production harness, this prompt should be invoked for a statistically sampled subset of traces—such as 5% of all routed requests or 100% of traces where the selected model exceeded a cost or latency threshold. The harness must supply the prompt with a structured trace payload containing the original user input, the classification scores, the routing rules that were active at the time, the selected model, and the available model catalog with cost and latency metadata. Without this structured input, the prompt cannot perform a meaningful audit.

The implementation should follow a strict validation-and-retry loop. After the LLM returns its audit result, the harness must validate the output against a defined schema: a JSON object containing a routing_decision_correct boolean, a recommended_model string, a cost_delta number, a latency_delta number, and an array of violations with rule_id and description fields. If validation fails, the harness should retry the prompt once with the validation error appended as a [CONSTRAINTS] update. If the second attempt also fails validation, the trace should be flagged for human review rather than silently accepted. For high-risk domains where routing errors could cause regulatory or safety issues, every audit result should require human approval before any routing policy change is deployed.

The harness should also log the full audit result alongside the trace ID, the prompt version, the model used for the audit itself, and the timestamp. This creates an evidence trail for governance reviews and allows trend analysis over time—for example, detecting that a specific routing rule is consistently flagged as suboptimal. Avoid wiring this prompt directly into the routing decision path in real time; it is an offline audit tool, not a synchronous gate. Adding it to the critical path would introduce latency and a new failure mode. Instead, use the audit output to update routing rules, adjust model weights, or tune classification thresholds in a subsequent deployment cycle.