Trace-Based SLO Threshold Evaluation Prompt Template

This prompt is designed for AI SREs and platform engineers who need to evaluate aggregated production trace data against defined Service Level Objectives (SLOs). Use it when you have a batch of trace metrics—latency percentiles, error rates, quality scores—and you need a structured breach evaluation with severity classification. This is not a prompt for raw trace inspection or single-trace debugging. It assumes you have already extracted and aggregated metrics from your observability pipeline and need the model to apply threshold logic, classify breach severity, and produce an evaluation report suitable for an alerting system or incident review.

The prompt works best when your SLO definitions are explicit and your input data is clean. The model acts as a deterministic evaluator, not an anomaly detector. It will not discover new failure modes; it will only assess whether known thresholds have been crossed. For this reason, you must provide a complete [SLO_DEFINITIONS] block with target values, evaluation windows, and severity mappings. The input [TRACE_METRICS] should be pre-aggregated—passing raw span data will produce unreliable results. If your metrics contain gaps or incomplete dimensions, the prompt includes a requirement to flag data quality issues rather than silently producing a clean evaluation.

Do not use this prompt for real-time alerting on individual traces; it is designed for batch evaluation over a defined window. For real-time use cases, deploy the threshold logic directly in your metrics pipeline and use this prompt only for periodic review or incident postmortem analysis. Avoid using this prompt when SLO definitions are still being negotiated—the model will apply whatever thresholds you provide without questioning their business validity. For threshold tuning and calibration, use the companion Alert Threshold Tuning Recommendation Prompt Template instead.

The following prompt template is designed to be pasted directly into your evaluation harness. It accepts structured trace data, SLO definitions, and evaluation parameters, then returns a structured threshold evaluation with breach classifications. All placeholders use square brackets and must be replaced with your actual data before execution. The template assumes you have already extracted relevant spans from your observability pipeline and normalized them into the expected input schema.

textCopy
You are an AI SRE evaluating production trace data against defined Service Level Objectives (SLOs).

Your task is to analyze the provided trace spans, compare them against the SLO thresholds, classify any breaches by severity, and return a structured evaluation.

## INPUTS

### Trace Data
[TRACE_SPANS_JSON]

### SLO Definitions
[SLO_DEFINITIONS_JSON]

### Evaluation Parameters
- Evaluation window: [EVALUATION_WINDOW]
- Current error budget remaining: [ERROR_BUDGET_REMAINING_PERCENT]
- Previous evaluation timestamp: [PREVIOUS_EVALUATION_TIMESTAMP]

## OUTPUT SCHEMA

Return a JSON object with this exact structure:
{
  "evaluation_id": "string",
  "evaluation_timestamp": "ISO8601",
  "window_start": "ISO8601",
  "window_end": "ISO8601",
  "slos_evaluated": [
    {
      "slo_name": "string",
      "metric": "latency_p95 | latency_p99 | error_rate | quality_score | token_efficiency",
      "threshold": number,
      "observed_value": number,
      "unit": "ms | percent | score",
      "breach_detected": boolean,
      "breach_severity": "critical | warning | none",
      "breach_magnitude_percent": number,
      "affected_spans": ["span_id_1", "span_id_2"],
      "contributing_factors": ["string"],
      "error_budget_consumed_percent": number
    }
  ],
  "overall_status": "healthy | degraded | critical",
  "requires_immediate_action": boolean,
  "recommended_actions": ["string"],
  "evaluation_confidence": number
}

## CONSTRAINTS

1. Classify breach severity using these rules:
   - **critical**: Observed value exceeds threshold by more than 50% OR error budget remaining is below 10%
   - **warning**: Observed value exceeds threshold by 10-50% OR error budget burn rate indicates exhaustion within [BURN_RATE_WINDOW_HOURS] hours
   - **none**: Observed value is within threshold

2. For latency SLOs, calculate percentiles from the provided span durations. If fewer than [MINIMUM_SAMPLE_COUNT] spans are available, set evaluation_confidence to "low" and note insufficient data.

3. For error rate SLOs, count spans with status "error" or "exception" against total spans in the window.

4. For quality score SLOs, use the [QUALITY_SCORE_FIELD] from span metadata. If absent, mark as unevaluable.

5. When multiple SLOs are breached, set overall_status to the highest severity among them.

6. Include specific span IDs in affected_spans to enable downstream drill-down.

7. Set evaluation_confidence between 0.0 and 1.0 based on data completeness, sample size, and measurement precision.

## EXAMPLES

Example Input:
Trace spans show 3 of 100 requests with latency > 500ms. SLO threshold is P95 < 500ms.

Example Output:
{
  "slo_name": "api-latency-p95",
  "metric": "latency_p95",
  "threshold": 500,
  "observed_value": 487,
  "unit": "ms",
  "breach_detected": false,
  "breach_severity": "none",
  "breach_magnitude_percent": 0,
  "affected_spans": [],
  "contributing_factors": [],
  "error_budget_consumed_percent": 0.5
}

## RISK LEVEL
[RISK_LEVEL]

If RISK_LEVEL is "high", append this to your response before the JSON: "HUMAN_REVIEW_REQUIRED: This evaluation contains critical breaches that may trigger incident response. A human SRE must validate before automated actions are taken."

To adapt this template for your environment, replace the trace data placeholder with spans exported from your observability platform in a consistent JSON format. Define your SLOs as a separate configuration object that maps metric names to thresholds, units, and evaluation rules. The evaluation window should align with your monitoring intervals—typically 5 minutes for real-time alerting or 1 hour for trend analysis. Before deploying, run this prompt against a set of known-good and known-breach traces to calibrate the severity classification rules against your team's incident response policies. If your risk level is high, ensure the human-review guardrail is wired into your alerting pipeline so that automated actions require explicit approval.

This prompt is designed to be called by an automated monitoring service, not a human chat interface. The typical integration pattern is a scheduled job (e.g., a cron-triggered Cloud Function, an Airflow DAG, or a Prometheus alertmanager webhook receiver) that queries your trace store for the relevant evaluation window, assembles the prompt with the current [TRACE_DATA], [SLO_DEFINITIONS], and [EVALUATION_WINDOW] parameters, and sends the request to the model. The output is parsed and routed to your incident management system (PagerDuty, Opsgenie) or observability platform (Datadog events, Grafana annotations) based on the breach_severity field. Do not use this prompt for real-time per-request decisions; it is designed for aggregate window evaluation where a 30-60 second model latency is acceptable.

Before the prompt is called, validate that [TRACE_DATA] contains all required span attributes: service.name, duration_ms, status.code, and any custom attributes referenced in your SLO definitions (e.g., groundedness_score, tool_call_success). If the trace store query returns an empty result set, skip the model call and emit a NO_DATA status to your monitoring system to avoid false-positive breach evaluations. After the model responds, apply a strict output schema validator (Pydantic or JSON Schema) to confirm the presence of slo_id, breach_detected, breach_severity, current_value, threshold, and evaluation_timestamp for each evaluated SLO. If validation fails, implement a single retry with a backoff of 2 seconds and a slightly lowered temperature (0.0 instead of 0.1) to reduce output variance. If the retry also fails, log the raw response and escalate to the on-call channel with a PROMPT_OUTPUT_INVALID alert rather than silently dropping the evaluation.

For high-severity breach classifications (critical or major), route directly to your incident management system with the full evaluation payload attached as evidence. For minor or warning severities, write the evaluation result to your observability dashboard as a non-paging event and increment a counter metric (slo_breach_warning_total) for trend tracking. Implement a deduplication window of 5 minutes per SLO to prevent alert storms when the same threshold is breached repeatedly within a short span. Finally, log every evaluation—including non-breach results—to a structured audit table with the prompt version hash, model identifier, and raw trace window boundaries. This audit log becomes your source of truth for tuning threshold sensitivity and defending SLO compliance to stakeholders.