Override Pattern Detection Prompt

This prompt is designed for a post-hoc, batch-analysis workflow where governance teams, audit leads, or compliance engineers need to surface statistically meaningful patterns from a collection of structured override records. The core job-to-be-done is transforming a raw dataset of human overrides—each with a justification, category, reviewer, and timestamp—into a structured summary of trends, anomalies, and potential policy gaps. It is not a real-time escalation tool; it belongs in a periodic review cadence (weekly, monthly, or per-quarter) or an on-demand investigation triggered by an anomaly flag. The ideal user is a technical auditor or compliance engineer who can query the override database, format the results, and critically evaluate the AI-generated analysis before it informs policy changes.

Before using this prompt, you must have a queryable, structured dataset ready. Required fields for each override record include a unique identifier, a timestamp, a reviewer or team identifier, a categorized override reason, and the original AI decision that was overridden. The prompt expects this data to be provided in a structured format, such as a JSON array or a markdown table, within the [OVERRIDE_RECORDS] placeholder. The analysis is only as good as the input data; if override reasons are uncategorized free-text, you must first apply the Override Reason Categorization Prompt to normalize the data. Do not use this prompt on a dataset smaller than 20-30 records, as the statistical pattern detection will produce unreliable, high-noise results that can mislead reviewers.

This prompt must never be used as an automated decision-maker. The output is a structured summary intended for human review, not a direct trigger for policy changes or remediation actions. Always pair this prompt with a human-in-the-loop step to validate detected patterns. A common failure mode is treating a detected correlation as a confirmed root cause; the prompt can identify that a specific team overrides more frequently, but it cannot know if that is due to poor AI performance, a risk-averse team culture, or a genuine gap in policy. Your next step after receiving the output should be to validate the top 2-3 detected patterns against the raw records and, if confirmed, draft a remediation hypothesis for stakeholder review.

The following prompt instructs the model to act as a governance analyst. It expects a batch of structured override records as input and produces a report identifying trends, anomalies, and potential policy gaps. The prompt is designed to be wired into an automated audit pipeline where it runs on a scheduled basis or is triggered by a new batch of records.

textCopy
You are a governance analyst reviewing a batch of human override records from an AI system. Your task is to detect patterns, anomalies, and potential policy gaps in the override data.

INPUT DATA:
[OVERRIDE_RECORDS]

ANALYSIS PARAMETERS:
- Time Window: [TIME_WINDOW]
- Grouping Dimensions: [GROUPING_DIMENSIONS]
- Anomaly Sensitivity: [ANOMALY_SENSITIVITY]
- Minimum Pattern Frequency: [MIN_PATTERN_FREQUENCY]

INSTRUCTIONS:
1. Parse the input records and validate that each record contains the required fields: override_id, timestamp, reviewer_id, team, decision_type, override_reason, original_ai_decision, and outcome.
2. Group records by the specified dimensions and analyze override frequency, reason distribution, and reviewer patterns.
3. Detect statistically significant trends (increasing, decreasing, or cyclical patterns) in override rates.
4. Identify anomalies such as sudden spikes, unusual reviewer behavior, or rare override reasons.
5. Flag potential policy gaps where override reasons suggest unclear guidelines, missing edge cases, or conflicting policies.
6. For each finding, provide supporting evidence by referencing specific record IDs.
7. Assign a severity level (LOW, MEDIUM, HIGH, CRITICAL) to each finding based on potential impact.

OUTPUT SCHEMA:
{
  "analysis_metadata": {
    "records_analyzed": int,
    "time_window": string,
    "analysis_timestamp": string
  },
  "summary_statistics": {
    "total_overrides": int,
    "override_rate": float,
    "unique_reviewers": int,
    "unique_teams": int,
    "top_reasons": [{"reason": string, "count": int, "percentage": float}]
  },
  "detected_patterns": [
    {
      "pattern_id": string,
      "pattern_type": "trend" | "cluster" | "correlation",
      "description": string,
      "supporting_record_ids": [string],
      "confidence": float,
      "severity": "LOW" | "MEDIUM" | "HIGH" | "CRITICAL"
    }
  ],
  "anomalies": [
    {
      "anomaly_id": string,
      "anomaly_type": string,
      "description": string,
      "affected_records": [string],
      "deviation_from_baseline": string,
      "severity": "LOW" | "MEDIUM" | "HIGH" | "CRITICAL"
    }
  ],
  "policy_gaps": [
    {
      "gap_id": string,
      "description": string,
      "related_pattern_ids": [string],
      "suggested_remediation": string,
      "urgency": "LOW" | "MEDIUM" | "HIGH"
    }
  ],
  "recommendations": [
    {
      "recommendation_id": string,
      "target": string,
      "action": string,
      "priority": int
    }
  ]
}

CONSTRAINTS:
- Only report patterns supported by at least [MIN_PATTERN_FREQUENCY] records.
- Flag low-confidence findings explicitly and do not treat them as actionable.
- Do not infer reviewer intent beyond what is stated in the override reason.
- If the input data is insufficient for analysis, return an error object with a clear explanation.
- Preserve record IDs exactly as provided for traceability.

To adapt this prompt for your environment, replace the square-bracket placeholders with live data before each run. The [OVERRIDE_RECORDS] placeholder should receive a JSON array of override objects. Set [ANOMALY_SENSITIVITY] to a value like "standard" or "high" to control how aggressively the model flags deviations. The [GROUPING_DIMENSIONS] field accepts a comma-separated list such as "team, decision_type, reviewer_id". Before deploying, validate that your override records conform to the expected schema and run this prompt against a labeled test set to calibrate the anomaly sensitivity and minimum pattern frequency parameters for your organization's override volume.

This prompt is designed for a batch processing pipeline, not a synchronous user-facing feature. The typical integration pattern involves a scheduled job (e.g., a nightly or weekly cron, Airflow DAG, or AWS Step Function) that queries a database or data warehouse for override records within a specified time window, assembles them into the [OVERRIDE_BATCH] input, and submits the batch to the LLM. The output is a structured pattern analysis that gets written to a governance dashboard, an audit log, or a compliance report. Because the analysis can influence policy changes, the harness must treat the LLM output as a draft recommendation, not a final determination.

The implementation should enforce a strict schema on the model's output using structured generation (e.g., JSON mode, function calling with a defined tool, or a Pydantic model parsed after generation). The expected output fields include detected_patterns (an array of pattern objects with pattern_name, description, supporting_evidence, and affected_records), anomalies (isolated outliers that don't form a pattern), statistical_significance_notes (caveats about sample size or confounding variables), and policy_gap_recommendations. After parsing, run a validation layer that checks: (1) every affected_record ID exists in the input batch, (2) pattern descriptions do not hallucinate reviewer names or dates not present in the source data, and (3) the output is not an empty or trivial analysis when the batch contains diverse records. If validation fails, retry once with a repair prompt that includes the validation errors. If the retry also fails, log the failure and route the batch to a human analyst queue with the raw override records and partial output attached.

Model choice matters here. Use a model with a large context window (at least 128K tokens) if batches exceed 50-100 override records, and prefer models with strong reasoning benchmarks for the pattern detection task. For high-stakes governance workflows, consider running the same batch through two different models and flagging batches where the pattern analyses diverge significantly for human review. Log every run with the input batch hash, model version, output, validation results, and reviewer identity if escalated. This creates an audit trail of the pattern detection process itself, which is essential when the analysis is used to justify policy changes or personnel actions. Avoid wiring this prompt directly into an automated enforcement system—pattern detection is diagnostic, not prescriptive, and should always leave room for human interpretation before action.