Escalation Fatigue Detection and Threshold Adjustment Prompt

This prompt is for operations engineers and AI platform teams who are responsible for the health of a human-in-the-loop review system. The job-to-be-done is to analyze historical escalation patterns to detect systemic over-escalation, signs of reviewer fatigue, and threshold drift that silently degrades decision quality over time. The ideal user is someone who already has access to a dataset containing escalation decisions, reviewer outcomes, and timestamps, and who needs to move from anecdotal evidence ('the queue feels noisy') to a data-driven adjustment plan. This is not a prompt for real-time triage of a single item; it is a periodic system health check, run weekly or after a significant change in volume, policy, or personnel.

Do not use this prompt when you need an immediate escalation decision on a live input. It is designed for offline analysis of aggregate patterns, not for inline classification. It is also inappropriate if you lack historical reviewer outcome data—the prompt's core value is in comparing the system's escalation decision against the human's eventual resolution to identify mismatches. If your review queue is new and has fewer than a few hundred resolved items, the statistical signals will be too weak to produce reliable recommendations. In that case, start with the 'Confidence Threshold Escalation Prompt Template' to establish a baseline before returning here for calibration.

The prompt requires several structured inputs to be effective: a time-bounded dataset of escalation events with fields for escalation_reason, reviewer_outcome, reviewer_response_time_minutes, and escalation_timestamp. You must also supply the current escalation thresholds or rules as [CURRENT_THRESHOLDS] so the model can identify drift. The output is an adjustment recommendation, not an automated change. Every recommendation must be reviewed by a human operator before being applied to a production system. The prompt includes explicit instructions to flag recommendations with low statistical confidence, preventing over-correction on sparse data. The next step after generating the analysis is to run the proposed threshold changes against a held-out historical period to validate that false-positive and false-negative rates move in the predicted direction.

The following prompt template is designed to be integrated into an automated monitoring pipeline. It accepts a structured log of recent escalations and their review outcomes, analyzes them for patterns indicative of reviewer fatigue or threshold drift, and produces a concrete set of adjustment recommendations. The template uses square-bracket placeholders for all dynamic inputs, allowing you to inject data from your logging database, ticketing system, or monitoring dashboard directly into the prompt.

textCopy
You are an operations analyst reviewing the health of a human-in-the-loop escalation system. Your goal is to detect signals of escalation fatigue, reviewer desensitization, or threshold drift, and to recommend specific, measurable adjustments.

## INPUT DATA
Review the following escalation log, which contains recent items escalated for human review and their final outcomes.
[ESCALATION_LOG]

## ANALYSIS INSTRUCTIONS
1. **Fatigue Signal Detection**: Analyze the log for patterns that indicate reviewer fatigue or desensitization. Look for:
   - Decreasing overturn rates over time (reviewers increasingly rubber-stamping automated decisions).
   - Decreasing time-to-resolution on complex items (rushing through reviews).
   - Increasing variance in decisions for similar item types (inconsistency).
   - Clustering of "Approved" outcomes late in a shift or queue batch.
2. **Threshold Drift Analysis**: Compare the original escalation criteria against actual review outcomes. Identify cases where:
   - Items are being escalated but consistently approved with no action taken (over-escalation).
   - Items with similar characteristics are not being escalated but later found to require intervention (under-escalation).
3. **Impact Projection**: For each identified issue, project the impact on:
   - False-positive rate (items escalated unnecessarily).
   - False-negative rate (items missed that should have been escalated).
   - Reviewer workload and estimated cost.

## OUTPUT FORMAT
Produce a JSON object with the following structure. Do not include any text outside the JSON object.
{
  "analysis_timestamp": "ISO 8601 timestamp of analysis",
  "data_window": {
    "start": "ISO 8601 timestamp of earliest log entry",
    "end": "ISO 8601 timestamp of latest log entry",
    "total_items_reviewed": <integer>
  },
  "fatigue_signals_detected": [
    {
      "signal_type": "decreasing_overturn_rate" | "decreasing_time_to_resolution" | "increasing_decision_variance" | "batch_end_approval_clustering",
      "severity": "low" | "medium" | "high" | "critical",
      "evidence_summary": "Specific data points from the log supporting this signal.",
      "affected_reviewer_ids": ["<string>"] or null
    }
  ],
  "threshold_drift_analysis": [
    {
      "drift_type": "over_escalation" | "under_escalation",
      "original_criteria": "The escalation rule or threshold that is drifting.",
      "evidence_summary": "Specific examples from the log illustrating the drift.",
      "estimated_false_positive_rate": <float between 0 and 1> or null,
      "estimated_false_negative_rate": <float between 0 and 1> or null
    }
  ],
  "adjustment_recommendations": [
    {
      "recommendation_id": "<string>",
      "target_criteria": "The specific escalation rule, threshold, or policy to adjust.",
      "current_value": "<string>",
      "proposed_value": "<string>",
      "rationale": "Why this adjustment addresses the detected fatigue or drift.",
      "projected_impact": {
        "false_positive_change": "<string describing expected change>",
        "false_negative_change": "<string describing expected change>",
        "workload_change": "<string describing expected change>"
      },
      "implementation_risk": "low" | "medium" | "high",
      "calibration_test_suggestion": "A specific test to validate this adjustment before full deployment."
    }
  ],
  "overall_system_health_assessment": "A concise summary of the escalation system's current health."
}

## CONSTRAINTS
- Base all analysis strictly on the provided [ESCALATION_LOG]. Do not invent or assume data points not present.
- If the log is insufficient to detect a specific signal, set its severity to 'low' and note the data limitation in the evidence_summary.
- Recommendations must be specific and measurable. Avoid vague advice like 'review the threshold'.
- If no adjustments are needed, return an empty array for adjustment_recommendations and explain why in the overall_system_health_assessment.

To adapt this template, replace the [ESCALATION_LOG] placeholder with a structured JSON array of escalation events. Each event should include a unique ID, timestamp, escalation reason, reviewer ID, review outcome, and time-to-resolution. For high-stakes production systems, implement a validation step that checks the output JSON against the defined schema before any automated threshold changes are applied. Always route high or critical severity findings and any recommendation with an implementation_risk of high for human review before enacting adjustments. Calibrate the prompt's sensitivity by running it against a historical dataset with known fatigue events to ensure it detects real issues without flooding your team with false alarms.

This prompt is designed to run as a scheduled batch job—not a real-time request handler. Wire it into a cron or workflow orchestrator (e.g., Airflow, Prefect, Temporal) that executes daily or weekly against a rolling window of escalation records. The input dataset should be pulled from your escalation log store (database, data warehouse, or log aggregator) and pre-aggregated into the [ESCALATION_METRICS] structure before the prompt is called. Do not pass raw event streams directly to the model; pre-compute counts, rates, and reviewer-level statistics in the application layer to keep the prompt focused on analysis rather than arithmetic.

Validation and retry logic is critical because the output drives threshold changes. After the model returns its JSON response, validate the adjustment_recommendations array against a schema that checks: (1) every recommended threshold value falls within your system's allowed range (e.g., 0.0–1.0 for confidence scores), (2) the predicted_impact fields contain numeric values for both false-positive and false-negative rate changes, and (3) the confidence score for each recommendation is present and between 0 and 1. If validation fails, retry once with the validation errors appended to the prompt as additional [CONSTRAINTS]. If the second attempt also fails, log the failure and escalate to the on-call channel—do not silently apply unvalidated threshold changes. For high-stakes production systems, add a human approval gate: write the validated recommendations to a review queue (e.g., a Slack message, Jira ticket, or internal dashboard) and require explicit approval before the new thresholds are pushed to your routing configuration.

Model choice matters. Use a model with strong reasoning capabilities (GPT-4o, Claude 3.5 Sonnet, or equivalent) because the task requires multi-variable trade-off analysis across false-positive rates, false-negative rates, and reviewer fatigue signals. Cheaper or smaller models tend to produce plausible-sounding but mathematically inconsistent recommendations—especially when the prompt includes calibration test cases. If you're running this at high frequency or against large datasets, consider caching the prompt prefix (the system instructions and output schema) to reduce token costs. Log every invocation—including the input metrics, the raw model output, validation results, and the final approved or rejected thresholds—to create an audit trail for governance reviews and to enable retrospective analysis when escalation quality drifts.

What to avoid: Do not apply threshold adjustments automatically without the calibration test step. The prompt includes [CALIBRATION_TESTS] for a reason—run them every time and reject recommendations that fail to correctly classify the test cases. Do not use this prompt on windows shorter than one week; fatigue patterns require sufficient data to distinguish signal from noise. Do not treat the model's recommendations as the final word on reviewer workload—combine the output with operational metrics like average review time, queue depth, and SLA breach rates before making staffing or process changes. Finally, if your escalation system uses multiple models or routing paths, run this analysis per path rather than on aggregate data to avoid masking path-specific fatigue patterns.