Error Budget Burn Trace Attribution Prompt

This prompt is designed for Site Reliability Engineers (SREs) and FinOps practitioners who are actively responding to a Service Level Objective (SLO) breach. The core job-to-be-done is to rapidly attribute error budget consumption across a set of production traces to specific, actionable failure categories—such as model errors, tool timeouts, refusal spikes, or latency overruns—and rank them by their impact on the error budget. The ideal user has access to a batch of traces from the breach window and needs a structured, evidence-backed report to guide remediation and communicate impact to stakeholders, rather than manually sifting through individual spans.

Use this prompt when you have already isolated a time window and a set of traces associated with a known SLO breach. It is most effective when the input traces contain structured metadata, including span-level error codes, latency measurements, and tool-call outcomes. The prompt requires a clear definition of your error budget rules and failure categories to produce a reliable attribution. Do not use this prompt for real-time alerting, for diagnosing a single failed request in isolation, or when the root cause is already known and does not require a ranked, quantitative breakdown. It is an analytical tool for post-hoc incident scoping, not a replacement for your monitoring dashboards.

Before executing this prompt, ensure your trace data is pre-processed to include the necessary fields: a unique trace ID, a timestamp, a failure classification tag, and a latency value. The prompt's value hinges on the quality of this input data. The output is a burn-down attribution report, which should be treated as a diagnostic aid, not a final verdict. For high-severity incidents, always pair the model's output with human review of the top attributed traces to confirm the failure signatures before communicating the report to stakeholders. The next step after generating this report is to use the ranked attribution to drive a fix-forward or rollback decision, often by feeding the top failure category into a more targeted diagnostic prompt, such as the 'Tool-Call Error Correlation Prompt Across Sessions' or the 'Latency Spike Root-Cause Trace Prompt.'

The following prompt template is designed to be copied directly into your observability or incident-response tooling. It accepts a batch of production traces from an SLO breach window and outputs a structured burn-down attribution report. Every placeholder is enclosed in square brackets and must be replaced with real data before execution. The template assumes you have already extracted relevant traces—filtered by the error budget burn period—and have them available in a structured format such as JSON span arrays or log entries.

codeCopy
You are an SRE trace analyst. Your task is to attribute error budget consumption across a set of production traces to specific failure categories and rank them by budget impact.

[INPUT]
A batch of production traces from the SLO breach window. Each trace includes:
- trace_id
- spans with operation_name, service, duration_ms, status (OK/ERROR), error_message
- model_request and model_response payloads where applicable
- tool_call records with tool_name, arguments, response, and duration_ms
- refusal flags and safety filter activations
- latency breakdowns (model_inference_ms, tool_call_ms, retrieval_ms, overhead_ms)
- timestamp

[CONSTRAINTS]
1. Classify each trace into exactly one primary failure category from the list below.
2. If a trace matches multiple categories, choose the most impactful (highest latency contribution or error severity).
3. Ignore traces where status is OK and latency is within SLO threshold.
4. For each category, compute:
   - trace_count: number of traces in that category
   - total_budget_burn_seconds: sum of latency over SLO threshold across those traces
   - budget_burn_percentage: percentage of total error budget consumed by this category
   - top_contributing_trace_ids: up to 5 trace IDs with the highest individual burn
   - representative_error_message: the most common error message or failure signature
5. Flag any trace where the failure cause is ambiguous with `human_review_required: true`.

[FAILURE_CATEGORIES]
- MODEL_ERROR: model returned an error, hallucinated critical fields, or produced unparseable output
- TOOL_TIMEOUT: a tool call exceeded its timeout and caused request failure or severe latency
- REFUSAL_SPIKE: model refused to answer a valid request, triggering fallback or error
- LATENCY_OVERRUN: model or tool latency exceeded SLO threshold without an explicit error
- CONTEXT_OVERFLOW: context window overflow caused evidence loss or degraded output
- RETRIEVAL_FAILURE: vector store or search returned empty, irrelevant, or error responses
- INFRA_ERROR: DNS, network, OOM, or dependency service failure
- PROMPT_REGRESSION: output format drift, instruction violation, or behavioral change from prior version
- UNKNOWN: failure does not match any above category (requires human review)

[OUTPUT_SCHEMA]
{
  "report": {
    "breach_window": {
      "start": "ISO timestamp",
      "end": "ISO timestamp",
      "total_traces_analyzed": int,
      "total_budget_burn_seconds": float,
      "slo_threshold_ms": int
    },
    "attribution": [
      {
        "rank": int,
        "category": "FAILURE_CATEGORY",
        "trace_count": int,
        "total_budget_burn_seconds": float,
        "budget_burn_percentage": float,
        "top_contributing_trace_ids": ["trace_id", ...],
        "representative_error_message": "string",
        "human_review_required": boolean
      }
    ],
    "ambiguous_traces": [
      {
        "trace_id": "string",
        "possible_categories": ["CATEGORY", ...],
        "reason": "string explaining ambiguity"
      }
    ],
    "summary": "One-paragraph narrative summarizing the primary budget burn driver and recommended next step."
  }
}

[EXAMPLES]
Example input trace (abbreviated):
{
  "trace_id": "abc123",
  "spans": [
    {"operation_name": "tool_call/search", "status": "ERROR", "error_message": "timeout after 5000ms", "duration_ms": 5100}
  ],
  "latency_breakdown": {"tool_call_ms": 5100, "model_inference_ms": 200}
}
Expected classification: TOOL_TIMEOUT

Example input trace (abbreviated):
{
  "trace_id": "def456",
  "spans": [
    {"operation_name": "model_inference", "status": "OK", "duration_ms": 8000}
  ],
  "latency_breakdown": {"model_inference_ms": 8000},
  "slo_threshold_ms": 5000
}
Expected classification: LATENCY_OVERRUN

[RISK_LEVEL]
HIGH. This prompt informs incident response and stakeholder communication. Incorrect attribution can lead to misdirected remediation efforts. Always review ambiguous traces manually before finalizing the report.

To adapt this template, replace [INPUT] with your actual trace data formatted as specified. If your trace schema differs, update the field descriptions in the input block and adjust the [OUTPUT_SCHEMA] to match your internal reporting format. The failure categories in [CONSTRAINTS] should be customized to your system's known failure modes—add or remove categories based on your service architecture. For teams using OpenTelemetry, map operation_name to your span names and status to OTel status codes. Before running this prompt in an incident, test it against a labeled set of historical traces to verify classification accuracy and budget-burn arithmetic. The human_review_required flag is critical: never automate the final report without reviewing ambiguous traces, especially when the error budget burn is severe and stakeholders are waiting for a root-cause summary.

This prompt is designed to run during an active SLO breach or a scheduled error-budget review. It expects a batch of production traces—typically pulled from your observability platform (Datadog, Honeycomb, LangSmith, etc.)—along with the current error budget status and the SLO window definition. The output is a structured burn-down attribution report, not a raw trace dump, so the harness must enforce the output schema before the report reaches an incident commander or FinOps stakeholder.

Wire the prompt into an incident-response runbook or a scheduled FinOps pipeline. At invocation time, assemble the [TRACE_BATCH] by querying your trace store for all requests within the SLO window that consumed error budget (e.g., HTTP 5xx, timeout, refusal, or custom failure signals). Include span-level detail: model name, tool calls, latency per span, error codes, and prompt version tags. Populate [SLO_CONTEXT] with the current budget remaining, the burn rate, the window duration, and the threshold that triggered the alert. Set [FAILURE_CATEGORIES] to your organization's taxonomy—model errors, tool timeouts, refusal spikes, latency overruns, context overflows—and map each trace span to a category before calling the prompt. If your trace store doesn't natively tag categories, run a lightweight classification step first or pass raw spans and let the prompt classify them. The [OUTPUT_SCHEMA] should enforce a ranked list of categories with budget-consumed percentages, exemplar trace IDs, and a confidence score per attribution. Validate the output against this schema immediately after generation; reject and retry if the JSON is malformed, if percentages don't sum to within 1% of the total budget consumed, or if any category lacks at least one supporting trace ID.

Model choice matters here. Use a model with strong structured-output capabilities and a context window large enough to hold your trace batch. For incidents involving more than 50 traces, consider pre-aggregating spans into per-category summaries before calling the prompt, or split the batch and merge results with a second synthesis call. Log every invocation: input trace IDs, the generated attribution report, the validator result, and any retries. This audit trail is essential for postmortem review and for defending budget-attribution decisions to stakeholders. If the prompt's confidence score for any top category falls below your threshold (e.g., 0.7), flag the report for human review before distribution. Never send an unvalidated attribution report to an incident channel or executive summary—misattributed budget burn can drive the wrong remediation and erode trust in your observability pipeline.

For recurring FinOps use, schedule this prompt to run at the end of each SLO window and store the output in a time-series database alongside your budget metrics. This lets you track category trends over time and detect slow-burn regressions that don't trigger a single-incident alert. When wiring into an incident runbook, make the prompt call blocking: the incident commander should not close the incident until the attribution report is validated and reviewed. If the prompt fails after three retries, escalate to the AI SRE on-call with the raw trace batch and the validator error log. Avoid running this prompt on traces that haven't been deduplicated—duplicate spans will inflate budget-consumed percentages and corrupt the ranking.