Model Routing Decision Eval Failure Audit Prompt

This prompt is for AI platform engineers and SREs who need to isolate whether a production evaluation failure originated at the routing layer. When a response fails an automated quality, safety, or accuracy check, the root cause can hide in many places: the prompt, the retrieved context, a tool call, or the model itself. This playbook focuses on one specific failure mode—the router selected a model that was poorly matched to the task characteristics, and that mismatch cascaded into the eval failure. Use it when you have trace data that captures the routing decision, the model that handled the request, the task profile, and the eval outcome. The prompt produces a structured audit comparing the actual model selected against the task requirements and identifies routing logic gaps that may have contributed to the failure.

To use this prompt effectively, you must supply trace-level routing metadata. At minimum, you need the router's input context (user query, task classification, estimated complexity), the routing decision (which model was chosen and why), the model's actual response, and the eval result with its rubric dimensions. Without this data, the prompt cannot distinguish a routing error from a prompt defect or a retrieval gap. The ideal input also includes the router's configuration at the time of the request—model availability, cost thresholds, latency targets, and any fallback logic that was active. If your observability pipeline does not capture routing decisions as structured trace events, instrument that first. A routing audit is only as good as the decision metadata it can inspect.

Do not use this prompt when the eval failure is already confirmed to be a prompt or retrieval issue, when the routing decision was made by a human rather than an automated system, or when you lack trace-level routing metadata. This prompt is also not a replacement for a full incident postmortem—it narrows the investigation to one specific hypothesis. If the audit clears the routing layer, your next step is to investigate the prompt, the retrieval pipeline, or the model behavior directly. Use the sibling playbooks for eval failure root-cause triage or retrieval gap diagnosis to continue the investigation.

The following prompt template is designed to be copied directly into your investigation environment. It forces a structured comparison between the routing decision, the task characteristics, and the eval failure signal. Before using it, ensure you have collected the full trace for the failed request, including the router's input context, the selected model, any fallback attempts, and the specific eval rubric that was violated. The prompt assumes you are investigating a single trace at a time; for batch analysis, wrap this in an application loop that iterates over a set of failed trace IDs.

textCopy
You are an AI platform reliability engineer auditing a model routing decision that resulted in an eval failure.

Your task is to determine whether the eval failure is attributable to a suboptimal routing choice, and if so, what change to the routing logic would have prevented it.

## INPUT DATA

### Trace Context
- Trace ID: [TRACE_ID]
- Timestamp: [TIMESTAMP]
- User Input Summary: [USER_INPUT_SUMMARY]
- Task Category: [TASK_CATEGORY]
- Task Complexity Indicators: [COMPLEXITY_INDICATORS]
- Expected Output Type: [EXPECTED_OUTPUT_TYPE]

### Routing Decision
- Routing Policy Version: [ROUTING_POLICY_VERSION]
- Router Input Features Used: [ROUTER_INPUT_FEATURES]
- Selected Model: [SELECTED_MODEL]
- Model Capability Profile: [MODEL_CAPABILITY_PROFILE]
- Alternative Models Considered: [ALTERNATIVE_MODELS]
- Routing Confidence Score: [ROUTING_CONFIDENCE_SCORE]
- Fallback Activated: [FALLBACK_ACTIVATED]
- Fallback Model (if any): [FALLBACK_MODEL]

### Eval Failure Details
- Failed Eval Dimension: [FAILED_EVAL_DIMENSION]
- Eval Rubric Excerpt: [EVAL_RUBRIC_EXCERPT]
- Eval Score: [EVAL_SCORE]
- Passing Threshold: [PASSING_THRESHOLD]
- Eval Judge Model: [EVAL_JUDGE_MODEL]
- Failure Severity: [FAILURE_SEVERITY]

### Model Output
- Actual Output (truncated to 500 words): [ACTUAL_OUTPUT]
- Expected Output Characteristics: [EXPECTED_OUTPUT_CHARACTERISTICS]

## ANALYSIS INSTRUCTIONS

1. **Task-Model Fit Assessment**: Compare the task category, complexity indicators, and expected output type against the selected model's capability profile. Identify any capability gaps that could explain the eval failure.

2. **Alternative Model Evaluation**: For each alternative model considered, assess whether it would have been a better fit for this specific task. Consider capability alignment, known limitations, and cost/latency trade-offs.

3. **Routing Logic Gap Identification**: Determine whether the routing policy's input features adequately captured the task characteristics that mattered for this eval dimension. Flag any missing features that would have changed the routing decision.

4. **Confidence vs. Outcome Analysis**: Compare the routing confidence score with the actual outcome. If confidence was high but the result failed, identify what signal the router missed.

5. **Fallback Effectiveness**: If a fallback was activated, assess whether the fallback model was appropriate and whether the fallback trigger conditions were correct.

6. **Root Cause Classification**: Classify the root cause into one of:
   - ROUTING_ERROR: A different model should have been selected
   - FEATURE_GAP: The router lacked features needed to make the right decision
   - POLICY_THRESHOLD: The routing policy thresholds need adjustment
   - MODEL_CAPABILITY_GAP: No available model could have passed this eval
   - EVAL_ISSUE: The eval failure is a false positive or rubric problem
   - UNRELATED: The eval failure is unrelated to model choice

## OUTPUT FORMAT

Return a JSON object with this exact schema:
{
  "trace_id": "string",
  "routing_decision_quality": "adequate|suboptimal|incorrect",
  "root_cause_classification": "ROUTING_ERROR|FEATURE_GAP|POLICY_THRESHOLD|MODEL_CAPABILITY_GAP|EVAL_ISSUE|UNRELATED",
  "task_model_fit": {
    "selected_model_suitability": "suitable|marginal|unsuitable",
    "capability_gaps": ["string"],
    "best_alternative_model": "string or null",
    "best_alternative_rationale": "string"
  },
  "routing_logic_issues": [
    {
      "issue_type": "missing_feature|incorrect_weight|threshold_error|fallback_error",
      "description": "string",
      "recommended_change": "string"
    }
  ],
  "confidence_assessment": {
    "router_confidence": number,
    "was_confidence_warranted": "yes|no|partially",
    "missed_signal": "string or null"
  },
  "recommendation": {
    "action": "adjust_routing_policy|add_router_feature|recalibrate_thresholds|update_model_capability_profile|investigate_eval|no_change",
    "priority": "critical|high|medium|low",
    "summary": "string",
    "expected_impact": "string"
  },
  "evidence_chain": [
    {
      "observation": "string",
      "source": "trace_event|routing_log|eval_output|model_output",
      "relevance": "string"
    }
  ]
}

## CONSTRAINTS
- Do not speculate beyond the provided trace data. Mark any inference as such.
- If the trace data is insufficient to reach a conclusion, set root_cause_classification to UNRELATED and explain what data is missing.
- Do not recommend model changes that violate cost or latency constraints unless those constraints are explicitly provided.
- Preserve all trace IDs and timestamps exactly as provided.

To adapt this template for your environment, replace each square-bracket placeholder with data extracted from your observability platform. The [ROUTER_INPUT_FEATURES] field should include the actual feature vector or metadata the router used, not just a description. For the [MODEL_CAPABILITY_PROFILE], include the documented strengths, weaknesses, and intended use cases for each model in your routing pool. If your routing system does not produce confidence scores, set [ROUTING_CONFIDENCE_SCORE] to null and the analysis will skip that section. The output schema is designed to be machine-readable so you can aggregate results across multiple investigations into a routing quality dashboard. Before acting on any recommendation classified as critical, validate the finding against at least two additional failed traces with similar characteristics to avoid overfitting to a single incident.

The Model Routing Decision Eval Failure Audit Prompt is designed to be invoked as part of a structured investigation workflow, not as a standalone chat. When an eval failure is detected in production, the platform should automatically enrich the trace with the routing decision metadata, the task characteristics snapshot, and the eval outcome before calling this prompt. The prompt expects a complete trace segment that includes the original user input, the routing classifier's output (model selected, confidence, reasoning), the actual model that handled the request, the task profile (complexity, domain, expected capabilities), and the eval result with dimension-level scores. Without all four inputs, the audit will produce incomplete or misleading conclusions.

Wire this prompt into your investigation pipeline as a post-eval enrichment step. After the eval system flags a failure, the harness should: (1) retrieve the trace segment from your observability store using the trace ID and span ID; (2) extract the routing decision payload from the classifier service logs; (3) capture the task characteristics that were available at routing time (input length, detected domain, complexity score, required tool set); (4) assemble the eval outcome with per-dimension scores and the failing rubric criteria; and (5) invoke the prompt with all four inputs in a single structured payload. Validate the prompt's output against a schema that requires a routing_gap_hypothesis (one of: wrong_model_capability, task_misclassification, confidence_threshold_error, fallback_incorrect, no_gap_found), a severity score (1-5), and an evidence_chain array linking each finding to a specific trace event. If the output fails schema validation, retry once with a stricter constraint instruction appended. Log every audit result to a dedicated routing_audit index in your observability platform, keyed by trace ID, so that routing logic engineers can query patterns over time.

For high-risk domains where routing errors could cause safety or compliance failures, add a human review gate before the audit finding is accepted. The harness should flag any audit result with severity >= 4 or routing_gap_hypothesis == 'wrong_model_capability' for manual review in a queue. The reviewer should see the original trace, the prompt's audit output, and a diff between the model that was selected and the model that should have been selected according to the task profile. Do not automatically retrain or adjust routing weights based on a single audit result; aggregate findings across at least 50 correlated failures before proposing routing logic changes. Avoid running this prompt on every eval failure indiscriminately—filter first for failures where the eval dimension is plausibly routing-sensitive (e.g., accuracy, grounding, tool-use correctness) rather than format or latency issues.