Monthly Trace Health Report Prompt Template

This prompt is designed for engineering managers and AI program leads who must report on production AI system health to leadership on a monthly cadence. It takes aggregated trace data from your observability platform and produces a structured report with trend analysis, failure mode rankings, and actionable recommendations. Use this when raw trace volumes are too large for manual review and stakeholders need a decision-ready summary that preserves diagnostic signal without overwhelming detail.

The ideal user has already run aggregation queries against their observability store (e.g., Datadog, Arize, LangSmith, or an internal trace warehouse) and has pre-computed metrics such as error counts by category, P95 latency by step, tool-call failure rates, and eval pass/fail ratios. The prompt expects these aggregated numbers as input, not raw trace JSON. If you feed it thousands of individual trace objects, it will hit context limits and produce unreliable summaries. Instead, use the Trace Review and Diagnosis prompts for single-trace debugging, or the Weekly AI Operations Briefing prompt for higher-frequency operational updates.

Do not use this prompt for real-time incident response or live diagnosis. It is a retrospective reporting tool, not an on-call replacement. The output is a prose report with tables and bullet points, not a structured data payload for automated dashboards. If you need a machine-readable summary for downstream systems, pair this prompt with the Structured Output and Schema Control playbook to enforce a JSON output schema. For regulated or high-risk domains, always require human review of the final report before distribution, and ground every claim in the trace metrics you provided as input.

This template is designed to be pasted directly into your AI harness. It expects pre-processed trace data as input and produces a structured JSON report suitable for stakeholder review. Every square-bracket placeholder must be replaced with actual data before execution. The prompt enforces statistical grounding, actionable insight density, and clear separation of evidence from interpretation.

textCopy
You are an AI observability analyst producing a Monthly Trace Health Report for engineering leadership.

## INPUT DATA
[TRACE_DATA]

## REPORT PARAMETERS
- Reporting Period: [REPORTING_MONTH] [REPORTING_YEAR]
- Previous Period for Comparison: [PREVIOUS_MONTH] [PREVIOUS_YEAR]
- System Components Under Review: [SYSTEM_COMPONENTS]
- SLA Thresholds: [SLA_THRESHOLDS]
- Business Impact Context: [BUSINESS_CONTEXT]

## OUTPUT SCHEMA
Return a JSON object with the following structure:
{
  "executive_summary": "string (max 150 words)",
  "key_metrics": {
    "total_traces": number,
    "success_rate": number (0-100),
    "p50_latency_ms": number,
    "p95_latency_ms": number,
    "p99_latency_ms": number,
    "error_budget_consumed_pct": number,
    "month_over_month_change": {
      "success_rate_delta": number,
      "p95_latency_delta_ms": number,
      "trace_volume_delta_pct": number
    }
  },
  "top_failure_modes": [
    {
      "category": "string",
      "frequency": number,
      "trend": "increasing" | "decreasing" | "stable",
      "example_trace_id": "string",
      "impact_assessment": "string"
    }
  ],
  "component_health": [
    {
      "component": "string",
      "status": "healthy" | "degraded" | "critical",
      "success_rate": number,
      "p95_latency_ms": number,
      "notable_changes": "string"
    }
  ],
  "anomalies_detected": [
    {
      "description": "string",
      "detection_date": "string",
      "severity": "low" | "medium" | "high" | "critical",
      "affected_trace_count": number,
      "hypothesis": "string"
    }
  ],
  "improvement_recommendations": [
    {
      "recommendation": "string",
      "target_metric": "string",
      "expected_impact": "string",
      "effort_estimate": "low" | "medium" | "high"
    }
  ],
  "data_quality_notes": "string (caveats about data completeness or reliability)"
}

## CONSTRAINTS
- Every metric must be derived from [TRACE_DATA]. Do not invent numbers.
- If a metric cannot be calculated from the provided data, set it to null and explain in data_quality_notes.
- Month-over-month deltas must use [PREVIOUS_MONTH] data. If unavailable, set deltas to null.
- Top failure modes must be ranked by frequency. Include no more than 5.
- Anomalies must reference specific trace IDs or time windows present in the data.
- Recommendations must be actionable and tied to specific metrics.
- Use "unknown" or "insufficient data" rather than guessing.
- The executive_summary must be suitable for a VP-level audience.

## EVALUATION CRITERIA
Before returning the output, verify:
1. All percentages are between 0 and 100.
2. Latency values are positive numbers.
3. Failure mode categories are mutually exclusive.
4. Component status assignments are consistent with their success rates.
5. Recommendations reference specific failure modes or anomalies from the report.
6. No placeholder text remains in the output.

To adapt this template, replace [TRACE_DATA] with a structured dump of your production traces—typically a JSON array of trace objects containing timestamps, component paths, latency breakdowns, status codes, error messages, and tool-call records. The [SYSTEM_COMPONENTS] placeholder should list the specific services, models, or pipelines you want evaluated. For [SLA_THRESHOLDS], provide your target success rate, p95 latency ceiling, and error budget allocation. If previous-month comparison data is unavailable, set [PREVIOUS_MONTH] and [PREVIOUS_YEAR] to null and the prompt will handle the gap gracefully. Always validate the output JSON against the schema before presenting it to stakeholders—automated schema validation is strongly recommended for this workflow.

This prompt is not a one-off chat interaction; it is a scheduled reporting job that must run reliably, produce auditable output, and fail gracefully when inputs are incomplete. The implementation harness wraps the LLM call in application logic that handles data assembly, validation, retry recovery, and delivery. For a monthly health report, the pipeline typically runs on the first business day of the month, pulling trace data from an observability store (e.g., Datadog, LangSmith, Arize, or an internal trace warehouse) and assembling the [TRACE_METRICS_JSON], [PREVIOUS_REPORT_SUMMARY], and [ORGANIZATIONAL_CONTEXT] placeholders before the prompt is rendered.

Data assembly and pre-processing is the first harness stage. The application must query the trace store for the target month's metrics: total requests, error rates, P50/P95/P99 latency, token consumption, tool-call success rates, eval pass rates, and refusal counts. These should be aggregated into a structured JSON payload matching the [TRACE_METRICS_JSON] schema the prompt expects. The harness should also fetch the previous month's report summary for trend comparison and any organizational context (team changes, model upgrades, infrastructure migrations) that occurred during the period. If any required data source is unavailable or returns null, the harness should abort with a clear error log rather than passing incomplete data to the model, which would produce a misleading report.

Validation and retry logic must operate at two levels. First, validate the assembled input data: check that metric values are within plausible ranges, that date windows are correct, and that required fields are present. Second, validate the model's output against the expected schema. The prompt requests a JSON structure with executive_summary, trend_analysis, top_failure_modes, improvement_recommendations, and data_quality_notes fields. The harness should parse the response, confirm all required fields exist, verify that numeric claims in the summary match the input metrics, and flag any hallucinated statistics not present in the source data. If validation fails, retry once with the same prompt plus the validation error message appended as a correction hint. If the second attempt also fails, escalate to a human reviewer with the raw trace data and both failed outputs attached.

Model selection and cost control are critical for a monthly pipeline that may process large trace volumes. Use a model with strong structured output capabilities and a context window large enough to hold the full metrics payload plus the prompt. GPT-4o or Claude 3.5 Sonnet are appropriate defaults; avoid smaller models that may hallucinate trend interpretations or fail to maintain statistical precision. If the trace data payload exceeds ~8K tokens, consider pre-summarizing raw trace samples before passing them to the report prompt, or use a two-stage pipeline where a first pass extracts representative examples and a second pass generates the report. Always log token usage, latency, and cost per run for budget tracking.

Human review and delivery complete the harness. For a monthly report going to leadership, the output should never be sent directly from the model to stakeholders without review. Route the validated output to a review queue where an engineering manager can confirm that trend interpretations are accurate, failure modes are correctly prioritized, and recommendations are actionable. The review interface should display the input metrics alongside the generated report so the reviewer can spot-check claims. After approval, the report can be delivered via email, Slack, or a dashboard. Store the approved report as the [PREVIOUS_REPORT_SUMMARY] for next month's run, closing the feedback loop. Log every run's input, output, validation results, review decision, and delivery status for auditability.