Quality Rubric Violation Trace-to-Eval Mapping Prompt

This playbook is for AI reliability engineers, quality engineers, and platform teams who have access to step-level trace data and need to produce a linked chain of evidence connecting an eval failure back to its root cause. The prompt assumes you already have the failed eval result, the rubric definition, and the full trace log. It does not replace initial triage or aggregate trend analysis. Use it when you need a precise, auditable mapping from a specific rubric violation to the retrieval step, tool call, context truncation, or generation event that triggered it.

Before using this prompt, confirm that you have isolated a single eval failure with a clear rubric dimension that was violated. The trace must include step-level attribution—such as retrieval calls, tool invocations, context window snapshots, and generation events—with timestamps or sequence IDs. If your observability tooling only captures final outputs without intermediate steps, this prompt will not produce reliable results. The prompt works best when the eval rubric is structured with explicit pass/fail criteria per dimension, not a single aggregate score. For example, a rubric with dimensions like citation_accuracy, completeness, and format_compliance allows the prompt to trace each violation independently.

Do not use this prompt for aggregate trend analysis, real-time alerting, or initial triage of an incident. It is designed for deep-dive investigation of a single failure, not for scanning hundreds of traces. If you need to correlate multiple failures across sessions, start with the sibling playbook for incident trace correlation. If you suspect the eval itself is flawed, use the LLM Judge Score Inconsistency Investigation prompt first. After running this prompt, you should have a clear root-cause statement and a specific trace event ID that you can hand off to the team responsible for the failing component—whether that is retrieval, tool configuration, context assembly, or model behavior.

This prompt template is the core diagnostic tool for closing the loop between a production eval failure and its root cause. It is designed to be pasted into your investigation notebook or wired into an automated diagnosis pipeline. The template forces a structured, step-by-step analysis that links a single rubric violation to a specific trace event—such as a bad retrieval step, a truncated context window, or an incorrect tool call—and demands evidence for every claim. This prevents the common failure mode of hand-waving root-cause analysis and ensures the output is a verifiable artifact that can be shared with the engineer responsible for the fix.

textCopy
You are a production AI diagnostician. Your task is to map a specific quality rubric violation to the precise trace event that caused it. You must produce a linked chain of evidence, not a summary of suspicions.

## INPUT DATA

### The Quality Rubric Violation
- **Rubric Rule:** [RUBRIC_RULE_DESCRIPTION]
- **Failed Eval Dimension:** [EVAL_DIMENSION_NAME]
- **Eval Score:** [EVAL_SCORE]
- **Eval Justification:** [EVAL_JUSTIFICATION]

### The Model's Final Output

[FINAL_OUTPUT]

codeCopy

### The Production Trace
```json
[TRACE_JSON]

CONSTRAINTS

• [CONSTRAINTS]

OUTPUT SCHEMA

You must output a single JSON object with the following structure:

jsonCopy
{
  "violation_summary": "A one-sentence restatement of the violation.",
  "root_cause_event": {
    "trace_step_id": "The ID of the trace step that is the primary cause.",
    "event_type": "The type of event (e.g., retrieval, tool_call, context_truncation, llm_generation).",
    "event_description": "A concise description of what happened at this step."
  },
  "causal_chain": [
    {
      "step_order": 1,
      "trace_step_id": "The first step in the chain of causation.",
      "what_happened": "What occurred at this step.",
      "how_it_contributed": "How this step contributed to the final violation.",
      "evidence": "Quote the relevant part of the trace or output."
    }
  ],
  "severity_assessment": "critical|major|minor",
  "recommended_action": "A specific, actionable recommendation for the team that owns the failing component.",
  "confidence": 0.0
}

INSTRUCTIONS

1. Parse the trace JSON to understand the sequence of events: retrievals, tool calls, context assembly, and final generation.
2. Identify the specific requirement in the rubric rule that was violated.
3. Trace backward from the violation in the final output, through each preceding trace step, to find the root cause event.
4. Construct a causal chain where each link is supported by direct evidence quoted from the trace or output.
5. If multiple factors contributed, identify the primary root cause and note contributing factors in the causal chain.
6. If the trace evidence is insufficient to establish a definitive root cause, set confidence low and state what additional data is needed.

To adapt this template, replace the square-bracket placeholders with data from your production systems. The [TRACE_JSON] placeholder expects a structured trace object that includes step-level attribution—at minimum, a sequence of events with unique IDs, types, and payloads. If your observability platform captures traces in a different format, pre-process them into a normalized JSON structure before insertion. The [CONSTRAINTS] placeholder is for adding domain-specific rules, such as ignoring certain known benign anomalies or prioritizing specific failure categories. For high-risk domains like healthcare or finance, always route the output for human review before taking action on the recommended fix.

After pasting the prompt, run it against a model with strong reasoning capabilities and a large context window, as the trace data can be verbose. Validate the output JSON against the schema before ingesting it into your incident management system. For automated pipelines, wrap this prompt in a retry loop with a validator that checks for the presence of all required fields and a confidence score above your threshold. Log every invocation, including the input trace ID, the output causal chain, and the reviewer's decision, to build an audit trail for future incident postmortems.

This prompt is designed for interactive investigation, not for a real-time production code path. The typical harness is a debugging notebook, an internal observability dashboard, or a Slack bot that on-call engineers invoke during an incident. The prompt expects structured trace data with step-level attribution and a specific eval rubric violation as input. Before wiring it into any tool, confirm that your trace storage can return the full event sequence for a single session, including retrieval steps, tool calls, context window contents, and the final model output. Without step-level attribution, the prompt cannot produce a linked chain of evidence.

When building the harness, validate the input trace payload against a schema before calling the model. At minimum, check that the trace contains a session_id, an array of events with step_type, timestamp, input, and output fields, and a violation object with rubric_id, rubric_description, and failed_output. Reject malformed traces with a clear error message rather than sending incomplete data to the model. For model selection, use a model with strong reasoning capabilities and a context window large enough to hold the full trace. If the trace exceeds the context window, implement a pre-processing step that filters events to those most relevant to the violated rubric dimension—for example, if the rubric concerns citation accuracy, retain retrieval, ranking, and generation steps while summarizing unrelated tool calls. Log every invocation with the session_id, rubric_id, the model's verdict, and the evidence chain for auditability. In regulated domains, require a human reviewer to confirm the model's finding before closing the investigation.

The most common failure mode in production is trace incompleteness. If your observability pipeline samples traces or drops intermediate steps, the prompt will produce a weak or incorrect evidence chain. Before relying on this prompt for incident response, run a dry-run audit across 50 recent eval failures to measure what percentage of traces contain sufficient step-level detail. If the answer is below 80%, invest in trace completeness before investing in investigation automation. Also avoid using this prompt as a substitute for eval rubric refinement—if the same rubric violation repeatedly traces back to an acceptable model behavior, the rubric itself may need adjustment rather than deeper investigation.