Post-Escalation Feedback Loop Prompt for Continuous Improvement

This prompt is designed for AI operations teams that have an active human-in-the-loop (HITL) review process and need to systematically close the feedback loop between automated escalations and human decisions. The core job-to-be-done is identifying systematic patterns in reviewer behavior—specifically over-escalation (the AI escalated, but the human reversed it), under-escalation (the AI did not escalate, but the human would have), and misclassification (the AI escalated for the wrong reason). The ideal user is an engineering lead or operations engineer responsible for tuning escalation thresholds, refining prompt instructions, or adjusting routing logic based on empirical evidence rather than intuition.

To use this prompt effectively, you must have a structured dataset of at least 50 reviewed escalation records. Each record should contain the original input, the AI's escalation reason, the escalation decision, and the human reviewer's final outcome. This minimum volume is not arbitrary; it ensures statistical significance when identifying patterns. Without it, the analysis will overfit to individual cases and produce unreliable recommendations. The prompt expects these records as structured input and will refuse to generate adjustment suggestions if the sample size is insufficient, instead recommending more data collection. You should also have access to the current escalation prompt or threshold configuration so the analysis can map identified patterns back to specific instructions or parameters that need changing.

Do not use this prompt for real-time decision-making or as a substitute for human review. It is an offline analysis tool, not a runtime classifier. It is also inappropriate when review data is sparse, inconsistently labeled, or when the escalation criteria have changed significantly mid-batch, as this introduces confounding variables that the prompt cannot disentangle. After running the analysis, the next step is to take the structured output—specific threshold adjustments, prompt rewrites, or routing changes—and test them in a staging environment against a holdout set of reviewed cases before deploying to production. Avoid the temptation to apply every suggestion at once; change one variable at a time and measure the impact on escalation accuracy.

This prompt template is designed to close the feedback loop on your escalation system. It takes a batch of human review decisions and their corresponding original escalation reasons, then analyzes them to identify systemic patterns of over-escalation, under-escalation, or misclassification. The output is a structured report with actionable suggestions for improving your prompt thresholds, routing logic, or classification criteria. Use this template as the core of a scheduled batch job or an on-demand analysis tool for your AI operations team.

textCopy
You are an AI operations analyst reviewing the performance of an automated escalation system. Your task is to compare human reviewer decisions against the original reasons for escalation to identify patterns and suggest concrete improvements.

## INPUT DATA
You will receive a JSON array of review records. Each record contains:
- `review_id`: A unique identifier for the review.
- `original_escalation_reason`: The structured reason the system provided for escalating this item to a human.
- `human_reviewer_decision`: The final decision made by the human reviewer (e.g., 'OVERTURNED', 'UPHELD', 'RE-CLASSIFIED').
- `human_reviewer_notes`: Any notes or re-classification details provided by the reviewer.

<REVIEW_DATA>
[REVIEW_DATA]
</REVIEW_DATA>

## ANALYSIS INSTRUCTIONS
1.  **Categorize Outcomes:** For each record, classify the human decision into one of three categories:
    - `OVERRIDE`: The human reversed the automated decision (e.g., de-escalated an item that was escalated, or escalated an item that was not).
    - `CONFIRM`: The human agreed with the automated decision.
    - `REFINE`: The human agreed with the need for action but changed the category, priority, or routing.

2.  **Identify Patterns:** Analyze the `OVERRIDE` and `REFINE` records to find recurring patterns. Group them by the original escalation reason and the human reviewer's notes. Look for:
    - **Over-Escalation:** Cases where the system escalated an item that a human deemed safe or routine. What signals triggered the false positive?
    - **Under-Escalation:** Cases where the system failed to escalate an item that a human later flagged as critical. (Note: This analysis is limited to items that *were* escalated, so look for patterns where the human upgraded the severity or added missed risk tags).
    - **Misclassification:** Cases where the system escalated for the wrong reason (e.g., escalated for 'PII' when the real issue was 'Abusive Content').

3.  **Suggest Improvements:** Based on the identified patterns, propose specific, actionable changes. These should be framed as modifications to the system's prompts, thresholds, or routing rules. For example:
    - "Increase the confidence threshold for the 'Negative Sentiment' escalation trigger from 0.7 to 0.85 to reduce false positives from sarcastic comments."
    - "Add a pre-check rule: Do not escalate for 'PII' if the detected entity is a publicly listed company phone number."
    - "Refine the prompt for 'Policy Violation' detection to better distinguish between a direct threat and a hypothetical discussion."

## OUTPUT FORMAT
Return your analysis as a JSON object with the following structure. Do not include any text outside the JSON object.
{
  "summary": {
    "total_reviews_analyzed": <integer>,
    "override_rate": <float, percentage of reviews where human decision was OVERRIDE>,
    "confirm_rate": <float, percentage of reviews where human decision was CONFIRM>,
    "refine_rate": <float, percentage of reviews where human decision was REFINE>
  },
  "pattern_analysis": [
    {
      "pattern_name": "A short, descriptive name for the pattern (e.g., 'Sarcasm Misclassified as Urgent Negative Sentiment').",
      "category": "OVER_ESCALATION | UNDER_ESCALATION | MISCLASSIFICATION",
      "frequency": <integer, number of times this pattern occurred>,
      "original_escalation_reason": "The escalation reason associated with this pattern.",
      "human_reviewer_feedback_summary": "A concise summary of what human reviewers consistently noted.",
      "suggested_improvement": {
        "target": "PROMPT | THRESHOLD | ROUTING_RULE | PRE_PROCESSING_STEP",
        "description": "A clear, actionable description of the change to implement.",
        "expected_impact": "A prediction of how this change will affect override and confirm rates."
      }
    }
  ],
  "overall_recommendations": [
    "A high-level, prioritized list of the top 3-5 changes to make to the system."
  ]
}

## CONSTRAINTS
- Base your analysis strictly on the provided <REVIEW_DATA>. Do not infer patterns that are not supported by the data.
- If the data set is too small to identify a statistically significant pattern, state this clearly in the `overall_recommendations` field and do not fabricate patterns.
- Focus on actionable improvements. Avoid vague suggestions like 'improve accuracy'.

To adapt this template, replace the [REVIEW_DATA] placeholder with your actual batch of review records. The input JSON structure is critical; ensure your application layer formats the data to match the expected review_id, original_escalation_reason, human_reviewer_decision, and human_reviewer_notes fields. For high-stakes systems, implement a validation step that checks the output JSON against the defined schema before accepting the analysis. If the model returns malformed JSON, use a retry recovery prompt with the schema and the raw output to attempt a repair. Always log the full prompt and response for auditability, and consider routing recommendations that involve threshold changes to a human for final approval before they are deployed to production.

This prompt is designed to run as a periodic batch analysis job, not a real-time request. It consumes structured records of human review decisions and the original escalation reasons, then produces a structured improvement report. The implementation harness must ensure that the input data is complete, the analysis is statistically grounded, and the output recommendations are reviewed by a human before any prompt or threshold changes are deployed to production.

Input Assembly: The prompt requires a batch of review records, each containing the original escalation reason, the human reviewer's final decision, and any override notes. These records should be pulled from your review queue database or logging system. The [REVIEW_BATCH] placeholder expects a JSON array of objects with fields like review_id, escalation_reason, reviewer_decision (e.g., UPHELD, OVERTURNED, RECLASSIFIED), and reviewer_notes. Before calling the model, validate that the batch size meets a minimum threshold (e.g., 50+ records) to ensure statistical significance. The [CONFIDENCE_THRESHOLD] and [MIN_SAMPLE_SIZE] variables should be set based on your operational risk tolerance—start with 0.95 and 30 respectively.

Model Choice and Tool Use: Use a model with strong reasoning capabilities and a large context window (e.g., GPT-4o, Claude 3.5 Sonnet) to process the batch in a single call. Enable structured output mode to enforce the [OUTPUT_SCHEMA], which should define fields for pattern_summary, over_escalation_rate, under_escalation_rate, statistically_significant_findings (boolean), recommended_adjustments (array of objects with target, current_value, suggested_value, and rationale), and requires_human_approval (boolean). If the batch is too large for a single context window, split it by escalation reason category and run the prompt once per category, then merge the results.

Validation and Guardrails: After the model returns, run a post-processing validation step. Check that all recommended adjustments include a rationale field and that the statistically_significant_findings flag is consistent with the reported rates and sample sizes. If the model recommends a threshold change, verify that the suggested value is within a sensible range (e.g., a confidence threshold between 0.5 and 1.0). Log every analysis run, including the input batch ID, the model's raw output, and the validation results, to an audit table for governance and future calibration.

Human-in-the-Loop Integration: This prompt's output must never trigger an automated deployment. Route the validated report to a human review queue for an operations engineer or AI system owner. The reviewer should confirm or reject each recommended adjustment, and their decision should be logged alongside the analysis. Only after human approval should prompt templates, routing thresholds, or escalation criteria be updated in your production configuration. This closes the feedback loop while maintaining a necessary safety boundary against automated drift.