Alert Threshold Tuning Recommendation Prompt Template

This prompt is designed for AI Site Reliability Engineers (SREs) and operations leads who need to move from manually adjusting alert thresholds to data-driven tuning. The core job-to-be-done is to analyze historical alert and incident data to produce a concrete, defensible recommendation for adjusting a specific alert threshold. The ideal user has access to a dataset containing alert firing records, their true-positive or false-positive classifications, and ideally, the Mean Time to Resolve (MTTR) or severity of any linked incidents. The prompt's value is in forcing a structured trade-off analysis between sensitivity (catching real problems) and specificity (reducing noise), rather than simply suggesting a new number.

You should use this prompt when you have a specific alert that is generating an unacceptable rate of false positives or false negatives, and you have the historical data to back up a change. The required context is not just the alert's current configuration, but a clean sample of its past behavior. This includes the alert's name, the current threshold value, a time-bound dataset of alert firings with their outcomes (e.g., 'true positive', 'false positive'), and the business cost of a missed incident. Without this historical outcome data, the prompt cannot perform a quantitative analysis and will fall back to general advice, which is not its intended purpose. The output is a detailed recommendation report, not a simple configuration change, making it suitable for a change advisory board review.

Do not use this prompt for real-time alerting decisions or as a substitute for a statistical anomaly detection system. It is an offline analytical tool for periodic threshold review. It is also inappropriate for greenfield alerts with no history, as the analysis depends entirely on past performance. For high-severity security or safety alerts where the cost of a false negative is catastrophic, the prompt's recommendation must be treated as an input to a human decision, not an automated control. The next step after generating a recommendation is to validate it against a holdout set of historical data not included in the prompt's input to ensure the projected alert volume and precision improvements hold true before implementing the change in production.

This prompt template is designed to be copied directly into your AI harness, observability pipeline, or internal tooling. It accepts structured historical alert and incident data and returns threshold adjustment recommendations with confidence intervals, sensitivity trade-offs, and projected alert volumes. Every placeholder is marked with square brackets and must be replaced with real data or configuration before execution. The template assumes you have already extracted false-positive rates, incident response outcomes, and current threshold values from your monitoring system.

textCopy
You are an AI SRE analyst specializing in alert threshold optimization. Your task is to analyze historical alert and incident data and produce threshold adjustment recommendations.

## INPUT DATA
[ALERT_HISTORY_JSON]

## CURRENT THRESHOLDS
[CURRENT_THRESHOLDS_JSON]

## INCIDENT RESPONSE OUTCOMES
[INCIDENT_OUTCOMES_JSON]

## CONSTRAINTS
- Maximum acceptable false-positive rate: [MAX_FALSE_POSITIVE_RATE]
- Minimum required recall (true positive rate): [MIN_RECALL]
- Evaluation window: [EVALUATION_WINDOW]
- Business hours multiplier: [BUSINESS_HOURS_MULTIPLIER]
- SLO target: [SLO_TARGET]

## OUTPUT SCHEMA
Return a JSON object with this exact structure:
{
  "recommendations": [
    {
      "alert_name": "string",
      "current_threshold": number,
      "recommended_threshold": number,
      "confidence_interval": {
        "lower": number,
        "upper": number,
        "confidence_level": number
      },
      "expected_false_positive_rate": number,
      "expected_recall": number,
      "projected_alert_volume": {
        "daily": number,
        "weekly": number,
        "monthly": number
      },
      "sensitivity_trade_off": "string (explain what is gained and lost)",
      "risk_level": "low|medium|high|critical",
      "requires_human_approval": boolean
    }
  ],
  "summary": {
    "total_alerts_analyzed": number,
    "alerts_recommended_for_change": number,
    "overall_false_positive_reduction_pct": number,
    "estimated_incident_miss_risk": "string",
    "implementation_priority_order": ["string (alert names in order)"]
  },
  "caveats": ["string (limitations, assumptions, data quality notes)"]
}

## ANALYSIS INSTRUCTIONS
1. For each alert in the input data, calculate the observed false-positive rate and recall using the incident response outcomes as ground truth.
2. Identify thresholds where the current false-positive rate exceeds [MAX_FALSE_POSITIVE_RATE] or recall falls below [MIN_RECALL].
3. Propose new threshold values that optimize the trade-off between false positives and missed incidents.
4. Compute confidence intervals using the historical data distribution. Flag any recommendation where the sample size is too small for reliable intervals.
5. Project alert volumes at the new threshold using historical event frequency and the [BUSINESS_HOURS_MULTIPLIER].
6. Assign a risk_level based on the potential impact of missed incidents at the recommended threshold.
7. Set requires_human_approval to true for any recommendation where risk_level is "high" or "critical", or where the confidence interval width exceeds 30% of the recommended value.

## VALIDATION RULES
- Do not recommend a threshold outside the observed data range without explicit justification in caveats.
- If historical data covers fewer than [MIN_DATA_POINTS] events for an alert, set confidence_interval to null and flag in caveats.
- Never recommend a threshold that would reduce recall below [MIN_RECALL].
- If no threshold satisfies both false-positive and recall constraints, recommend the best achievable trade-off and mark risk_level as "critical".

To adapt this template for your environment, replace each placeholder with data from your monitoring system. The [ALERT_HISTORY_JSON] should contain per-alert time series with triggered timestamps, threshold values at trigger time, and whether each alert was a true or false positive. The [INCIDENT_OUTCOMES_JSON] must link incidents to the alerts that should have fired, enabling recall calculation. If your observability stack does not capture incident-to-alert mappings, you will need to enrich the data before using this prompt. Start with a single high-noise alert to validate the output shape before scaling to your full alert inventory. Always run the recommendations through the human approval gate specified in the output schema before applying changes to production thresholds.

The Alert Threshold Tuning Recommendation prompt is designed to sit inside an automated analysis pipeline, not as a one-off chat interaction. It expects structured historical data—alert events with timestamps, severity labels, incident outcomes, and false-positive flags—and produces threshold adjustment recommendations with confidence intervals and projected alert volumes. The implementation harness must validate both the input data shape and the output recommendation structure before any threshold change is applied to production monitoring systems.

Wire the prompt into a scheduled job or an on-demand SRE tool that fetches historical alert data from your observability platform (Datadog, Grafana, Prometheus Alertmanager) and incident data from your incident management system (PagerDuty, Opsgenie). Before calling the LLM, validate that the input payload contains the required fields: alert name, evaluation window, historical alert counts, false-positive rates, incident correlation flags, and current threshold values. If any required field is missing or malformed, abort the run and log a structured error rather than sending incomplete data to the model. After receiving the model's response, parse the JSON output and validate it against an expected schema: each recommendation must include a metric_name, current_threshold, proposed_threshold, confidence_interval_lower, confidence_interval_upper, expected_alert_volume_change_pct, false_positive_rate_impact, sensitivity_tradeoff_notes, and a risk_level enum of low, medium, or high. Reject any recommendation where the confidence interval is inverted (lower > upper) or where the proposed threshold falls outside the interval bounds. Log all validation failures with the raw model output for debugging.

For high-risk recommendations—those with risk_level: high, proposed threshold changes greater than 30%, or expected alert volume reductions above 50%—route the output to a human approval queue rather than applying changes automatically. Use a review interface that displays the original alert history summary, the model's trade-off analysis, and a diff view of current versus proposed thresholds. Only after explicit human approval should the threshold be pushed to your monitoring configuration via API or infrastructure-as-code update. For low and medium risk recommendations, you can apply changes automatically but must log the full recommendation payload, the approving run ID, and a rollback threshold snapshot. Implement a mandatory observation period (e.g., 7 days) after any automated threshold change, with a separate monitoring rule that alerts if the new threshold generates a false-positive spike or misses a real incident that the old threshold would have caught. This closed-loop validation ensures the prompt's recommendations are continuously evaluated against production outcomes, feeding back into future tuning cycles.