Latency and Cost Trend Report Prompt Template

This prompt is designed for FinOps engineers and infrastructure leads who need to produce a structured trend analysis of per-step latency and token costs from production AI traces for budget owners. The core job-to-be-done is translating raw, granular trace data from an observability platform into a decision-ready report that explains cost variance, identifies anomalies, and attributes spend to specific prompt versions, tool calls, or context-window expansions. The ideal user has access to aggregated trace data that includes per-step timing and token counts, and they need to communicate these findings to stakeholders who control budgets but may not understand the underlying AI infrastructure.

Use this prompt when you have a batch of production traces that span a meaningful time window—such as a week or a month—and you need to surface trends, not individual failures. The required inputs include a structured dataset with fields like trace_id, timestamp, step_name, latency_ms, input_tokens, output_tokens, model_id, prompt_version, and tool_calls. The expected output is a narrative report with sections for an executive summary, cost and latency trends, anomaly flags, and attribution breakdowns. The prompt is designed to be wired into a reporting pipeline where trace data is pre-aggregated and validated before being passed to the model, ensuring the LLM focuses on analysis rather than raw data cleaning.

Do not use this prompt for real-time alerting, single-trace debugging, or when the input data lacks per-step timing and token counts. It is not a replacement for a monitoring dashboard or an incident response runbook. If the data is incomplete or the cost attribution logic is not pre-validated, the model will produce a fluent but inaccurate report. Always run the output through eval checks for cost attribution accuracy and anomaly flagging before sharing it with budget owners. The next step after generating the report is to review the anomaly flags, verify them against the source data, and prepare a brief for stakeholders that includes clear recommendations for cost optimization.

This prompt template is designed to be copied directly into your AI harness and adapted for your specific observability stack. It accepts structured trace data, cost attribution records, and anomaly flags as inputs, and produces a stakeholder-ready trend report with variance explanations. Replace each square-bracket placeholder with data from your monitoring pipeline before sending the prompt to the model. The template assumes you have already aggregated per-step latency measurements, token consumption counts, and cost calculations from your production traces—this prompt does not perform raw trace analysis but rather synthesizes pre-processed metrics into a narrative report.

textCopy
You are a FinOps analyst producing a latency and cost trend report for budget owners and infrastructure leads.

Your task is to analyze the provided production trace metrics and produce a structured trend report that explains cost and latency patterns, identifies anomalies, and provides actionable variance explanations.

## INPUT DATA

### Trace Metrics
[TIMESERIES_DATA]

### Cost Attribution
[COST_BREAKDOWN]

### Anomaly Flags
[ANOMALY_EVENTS]

### Comparison Baseline
[BASELINE_METRICS]

### Time Period
[REPORTING_PERIOD]

## OUTPUT SCHEMA

Produce a JSON object with the following structure:

{
  "report_metadata": {
    "period": "string",
    "generated_at": "ISO8601 timestamp",
    "data_freshness": "ISO8601 timestamp of last trace ingested"
  },
  "executive_summary": {
    "headline": "One-sentence summary of the most important trend",
    "key_findings": ["Finding 1", "Finding 2", "Finding 3"],
    "cost_risk_level": "low | medium | high | critical",
    "latency_risk_level": "low | medium | high | critical"
  },
  "cost_analysis": {
    "total_cost": "number with currency",
    "cost_change_percent": "percentage change from baseline",
    "cost_by_step": [
      {
        "step_name": "string",
        "cost": "number",
        "percent_of_total": "number",
        "change_from_baseline_percent": "number",
        "variance_explanation": "string explaining why this step changed"
      }
    ],
    "cost_by_model": [
      {
        "model_name": "string",
        "cost": "number",
        "tokens_consumed": "number",
        "percent_of_total": "number"
      }
    ],
    "cost_anomalies": [
      {
        "anomaly_description": "string",
        "detected_at": "ISO8601 timestamp",
        "estimated_cost_impact": "number",
        "likely_root_cause": "string",
        "recommended_action": "string"
      }
    ]
  },
  "latency_analysis": {
    "p50_latency_ms": "number",
    "p95_latency_ms": "number",
    "p99_latency_ms": "number",
    "latency_change_percent": "percentage change from baseline",
    "latency_by_step": [
      {
        "step_name": "string",
        "p50_ms": "number",
        "p95_ms": "number",
        "change_from_baseline_percent": "number",
        "variance_explanation": "string"
      }
    ],
    "latency_anomalies": [
      {
        "anomaly_description": "string",
        "detected_at": "ISO8601 timestamp",
        "duration_impact_ms": "number",
        "likely_root_cause": "string",
        "recommended_action": "string"
      }
    ]
  },
  "trend_analysis": {
    "cost_trend": "increasing | decreasing | stable",
    "latency_trend": "increasing | decreasing | stable",
    "correlation_notes": "string describing any correlation between cost and latency changes",
    "projection_next_period": "string describing expected trends if no action is taken"
  },
  "recommendations": [
    {
      "priority": 1,
      "category": "cost | latency | both",
      "recommendation": "string",
      "expected_impact": "string",
      "effort_level": "low | medium | high"
    }
  ],
  "data_quality_notes": {
    "completeness": "percentage of expected traces present",
    "gaps": ["description of any missing data"],
    "confidence_level": "high | medium | low"
  }
}

## CONSTRAINTS

- Only use data provided in the input sections. Do not invent metrics or trends.
- When a variance explanation cannot be determined from the data, state "Insufficient data to determine root cause" rather than speculating.
- Flag any anomaly where cost or latency exceeds 2 standard deviations from the baseline.
- If the data quality notes indicate less than 80% completeness, set confidence_level to "low" and note this in the executive summary.
- Round all monetary values to two decimal places. Round latency values to the nearest millisecond.
- For percentage changes, use the format "+X.X%" for increases and "-X.X%" for decreases.
- If no anomalies are detected, return empty arrays for cost_anomalies and latency_anomalies.
- Prioritize recommendations by estimated cost or latency impact.

## EXAMPLES

### Example Variance Explanation
Input: Embedding step cost increased 15% while token count remained stable.
Output: "Embedding cost increase of 15% with stable token consumption suggests a model pricing change or routing to a higher-cost embedding model. Verify model routing configuration for the embedding step."

### Example Anomaly Description
Input: P99 latency spiked to 4500ms at 14:32 UTC on 2025-03-15, baseline P99 is 1200ms.
Output: "P99 latency spike to 4500ms (baseline: 1200ms) detected at 14:32 UTC on 2025-03-15. Correlated with a 3x increase in context window size for the generation step during the same period."

## RISK LEVEL
[RISK_LEVEL]

If RISK_LEVEL is "high" or "critical", include an additional field in the executive summary called "immediate_actions_required" with an array of actions that must be taken within 24 hours.

To adapt this template for your environment, replace [TIMESERIES_DATA] with your aggregated latency and token metrics in a structured format such as JSON or CSV. The [COST_BREAKDOWN] placeholder expects per-step and per-model cost attribution data from your FinOps pipeline. [ANOMALY_EVENTS] should contain any pre-detected anomalies from your monitoring system—this prompt will explain and contextualize them rather than detect new ones. [BASELINE_METRICS] provides the comparison point for trend calculations, typically the previous reporting period or a rolling average. Set [RISK_LEVEL] based on your organization's thresholds: use "high" when cost or latency exceeds warning thresholds, and "critical" when SLOs are breached. The output schema is designed to be machine-readable for dashboard integration while remaining human-readable for stakeholder review. Before deploying, test this prompt with at least three periods of historical data where you know the ground-truth trends, and validate that the generated variance explanations match your manual analysis.

This prompt is designed to be integrated into a scheduled reporting pipeline, not run as a one-off chat. The typical harness is a script or lightweight service that queries your observability backend (e.g., a trace database, log analytics platform, or cost monitoring API) for a defined time window, aggregates the raw trace data into the [TRACE_DATA] input structure, invokes the model, validates the output, and routes the final report to a stakeholder channel. The model call itself is the transformation step between raw operational data and a human-readable trend report.

Input assembly: The [TRACE_DATA] placeholder expects a pre-aggregated JSON object containing per-step latency percentiles (p50, p95, p99), token counts (input, output, total), and estimated cost, grouped by operation (e.g., tool_call:search, llm:gpt-4o, retrieval:vector_lookup). Do not pass raw span logs directly to the model. Instead, use a pre-processing step to compute these aggregates and variance metrics (e.g., standard deviation, week-over-week delta) before populating the template. The [COMPARISON_WINDOW] placeholder should be filled with a human-readable label like "previous 7 days" or "baseline week of Oct 1-7" to ground the trend analysis. Model choice: Use a model with strong structured output support and a large enough context window to hold the aggregated data. GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro are suitable. Avoid small or fast models that may hallucinate variance explanations or misattribute cost drivers. Output validation: The prompt instructs the model to return a JSON object matching [OUTPUT_SCHEMA]. Your harness must parse this JSON and run the following checks before the report is delivered: (1) verify that all cost attribution percentages sum to 100% (±1% tolerance), (2) confirm that flagged anomalies reference specific steps and metrics from the input data, not invented data points, and (3) check that the executive_summary field is under 150 words. If validation fails, re-prompt the model with the specific validation error and the original input, or fall back to a human review queue if the failure persists after two retries.

Logging and audit: Log the full prompt, model response, validation results, and final delivered report for each run. This is critical for FinOps and infrastructure reporting because cost attribution disputes or anomaly explanations must be traceable back to the source data and model reasoning. Store the logs in a system that supports search by time window and report ID. Human review gating: For reports that will be sent to budget owners or executives, add a human approval step before delivery. The harness should route the validated report to a review queue (e.g., Slack, email, or an internal dashboard) where an engineering lead can confirm that anomaly flags are reasonable and cost explanations align with known operational events. Once approved, the harness can format the JSON into a readable document (PDF, Confluence page, or email) and distribute it. Failure modes to anticipate: The most common production failure is stale or incomplete trace data producing a report that looks plausible but is wrong. Mitigate this by adding a data freshness check in the pre-processing step—if the latest trace timestamp is older than the reporting window, abort the run and alert the operator. A second failure mode is the model generating generic variance explanations (e.g., "increased traffic") when the real cause is a specific infrastructure change. Combat this by including known deployment events or change logs in an optional [CONTEXT] field so the model can correlate anomalies with operational changes.