Human Approval Rate Monitoring Prompt

Use this prompt when you have a log of AI decisions and their corresponding human review outcomes—approved, overridden, rejected—and you need to produce a formal monitoring specification. This specification should define how to calculate approval rates, detect statistically significant trends, trigger alerts when rates cross thresholds, and recommend whether to advance, hold, or revert the system's autonomy stage. The ideal user is a product manager or AI reliability engineer implementing a graduated autonomy program who already collects structured review data and needs the analytical rules to interpret it, not the raw data pipeline itself.

This prompt is designed for offline monitoring specification generation. It assumes you have a dataset with fields like decision_id, timestamp, ai_decision, human_outcome, and autonomy_stage. The prompt will produce a specification document that includes rate calculation formulas, statistical significance tests for low-volume systems, threshold definitions for alerting, and stage-adjustment recommendation logic. You should not use this prompt for real-time action approval decisions—it does not make approval choices, it defines the monitoring rules that detect when approval patterns change. It also should not be used when you lack structured review data or when the review outcomes are not clearly categorized into approved, overridden, and rejected states.

Before using this prompt, ensure you have defined your autonomy stages and have at least a representative sample of review data to inform threshold setting. The output specification should be treated as a draft that requires human review, particularly for the threshold values and alerting rules, since these carry operational risk. Once generated, the specification can be implemented in your monitoring stack, but the prompt itself is a design tool, not a runtime component. Avoid using this prompt for systems where human review is ad-hoc or unstructured, as the specification will lack the data foundation it needs to be meaningful.

This prompt template is the core of the Human Approval Rate Monitoring playbook. It instructs the model to act as an AI reliability analyst and produce a complete monitoring specification. The specification is designed to be handed off to an engineering team for implementation in a metrics dashboard or observability platform. The prompt forces the model to define precise calculations, statistical checks for low-volume systems, and actionable threshold recommendations, moving beyond a simple rate calculation to a full operational plan.

textCopy
You are an AI reliability analyst designing a monitoring system for human-in-the-loop AI workflows. Your task is to produce a complete monitoring specification for tracking the Human Approval Rate (HAR) as a signal for graduated autonomy readiness.

## Input Data
- **System Description:** [SYSTEM_DESCRIPTION]
- **Action Categories:** [ACTION_CATEGORIES]
- **Current Autonomy Stage:** [CURRENT_AUTONOMY_STAGE]
- **Target Autonomy Stage:** [TARGET_AUTONOMY_STAGE]
- **Historical Log Schema:** [LOG_SCHEMA]
- **Volume Context:** [VOLUME_CONTEXT]

## Output Schema
Produce a JSON object with the following structure:
{
  "monitoring_specification": {
    "primary_metric": {
      "name": "Human Approval Rate (HAR)",
      "calculation": "string describing the exact formula",
      "numerator_definition": "string",
      "denominator_definition": "string",
      "unit": "percentage"
    },
    "secondary_metrics": [
      {
        "name": "string",
        "calculation": "string",
        "purpose": "string"
      }
    ],
    "trend_analysis_rules": {
      "window_size_days": number,
      "minimum_data_points_for_trend": number,
      "trend_detection_method": "string describing method, e.g., '7-day rolling average slope'",
      "seasonality_handling": "string"
    },
    "statistical_significance_checks": {
      "low_volume_threshold": "string describing when volume is considered low",
      "method_for_low_volume": "string describing the statistical test or rule, e.g., 'Bayesian prior with Beta distribution'",
      "confidence_interval_required": "boolean",
      "p_value_threshold_if_applicable": number
    },
    "threshold_alerts": [
      {
        "alert_name": "string",
        "condition": "string describing the exact metric condition",
        "severity": "INFO | WARNING | CRITICAL",
        "recommended_action": "string"
      }
    ],
    "stage_adjustment_recommendations": {
      "promotion_criteria": "string describing the HAR conditions for advancing to [TARGET_AUTONOMY_STAGE]",
      "demotion_criteria": "string describing the HAR conditions for reverting to a lower stage",
      "hold_criteria": "string"
    },
    "implementation_notes": "string with advice on logging requirements, metric collection, and dashboard design"
  }
}

## Constraints
- All calculations must be unambiguous and implementable in SQL or a standard metrics system.
- Statistical methods must be appropriate for the [VOLUME_CONTEXT] provided.
- Thresholds must be justified in the context of the risk associated with [ACTION_CATEGORIES].
- Do not invent data. If information is missing to define a calculation, state the assumption clearly in the `implementation_notes`.
- The `stage_adjustment_recommendations` must directly reference the [CURRENT_AUTONOMY_STAGE] and [TARGET_AUTONOMY_STAGE].

To adapt this template, replace the square-bracket placeholders with your system's specifics. For [LOG_SCHEMA], provide the exact fields available in your decision logs, such as timestamp, action_type, ai_decision, human_decision, and reviewer_id. The [VOLUME_CONTEXT] is critical; a system handling 10 actions/day needs different statistical handling than one processing 10,000/day. After pasting the prompt and running it, validate the output JSON against the schema. The implementation_notes field is your direct line to the engineering team—ensure it contains concrete, actionable guidance before you consider the specification complete.

This prompt is designed to be called on a recurring schedule—daily, weekly, or per deployment cycle—by a monitoring service, not triggered by individual user requests. The harness should fetch raw approval/rejection event logs from your application database or event stream, aggregate them into the [APPROVAL_LOG_DATA] structure, and pass them to the model. Because the output is a structured monitoring specification (JSON with rate calculations, trend analysis, and recommendations), the harness must validate that the returned JSON conforms to the expected schema before any downstream action is taken. A schema validator should check for required fields like calculated_approval_rate, trend_direction, statistical_significance, and stage_adjustment_recommendation. If validation fails, retry once with a stricter prompt variant that includes the validation error message; if it fails again, log the raw output and alert a human operator.

The most critical integration point is the threshold alerting system. Parse the threshold_alerts array from the model output and feed each alert into your existing alerting infrastructure (PagerDuty, OpsGenie, or a custom webhook). Each alert should include the alert_level, metric_name, current_value, threshold_value, and the recommendation text. For low-volume systems where statistical_significance is flagged as false, suppress automated stage-adjustment actions and instead route the output to a human reviewer with a clear warning that the sample size is insufficient for reliable conclusions. The stage_adjustment_recommendation field should only be acted upon automatically if statistical_significance is true and the confidence_level exceeds your organization's pre-defined threshold (typically 95%). Otherwise, treat it as a human-review recommendation.

Log every invocation's input parameters, raw model response, validation result, and any triggered alerts to an immutable audit store. This is essential for governance and for debugging false-positive alert storms. When wiring this into a CI/CD pipeline for staged autonomy releases, use the recommended_action field as a gate input: if the recommendation is REVERT or HOLD, block the promotion pipeline step and notify the release manager. Avoid the temptation to auto-tune thresholds based on the model's output without human review of the first several cycles; use the initial runs to calibrate your alerting thresholds against observed baseline behavior before closing the loop.