Override Reason Consistency Scoring Prompt

Governance leads and audit teams use this prompt to measure whether human override justifications are applied consistently across teams, reviewers, and similar decision types. The core job-to-be-done is detecting unexplained variance in override reasoning before it becomes a regulatory or fairness liability. You should use this prompt when you have a batch of override records—each containing the original AI decision, the human override action, the free-text justification, and relevant metadata such as reviewer identity, team, and decision type—and you need a quantitative consistency score plus specific divergence examples to prioritize investigation.

This prompt is designed for offline batch analysis, not real-time decision-making. Do not use it for approving or rejecting a single override in flight, for evaluating the sufficiency of one justification in isolation, or for making automated policy enforcement decisions. The prompt normalizes for legitimate contextual differences—such as different risk levels or policy sections—while flagging unexplained variance that may indicate inconsistent policy application, reviewer bias, or gaps in override training. The output includes a consistency score, a summary of normalization factors applied, and a list of specific override pairs or clusters that diverge significantly from the norm.

Before running this prompt, ensure your override records are complete and normalized into a consistent schema. Missing justification fields, inconsistent decision-type taxonomies, or unlabeled reviewer teams will degrade the analysis and produce unreliable scores. After receiving the output, treat the consistency score as an investigative lead rather than a final judgment. Human auditors should review the divergence examples to determine whether the variance is justified by undocumented context or represents a genuine policy gap. For high-stakes audits, pair this prompt with the Override Decision Comparison Prompt to drill into specific divergent pairs and the Override Justification Sufficiency Evaluation Prompt to assess whether individual justifications meet minimum standards.

The following prompt template is designed to be copied directly into your AI harness. It instructs the model to analyze a batch of override justification records and return a consistency score along with specific divergence examples. The template uses square-bracket placeholders that you must replace with your actual data, policy context, and desired output format before use. This prompt assumes you have already extracted and normalized override records into a structured format; it does not perform extraction from free-text logs or chat transcripts.

textCopy
You are an audit governance analyst. Your task is to evaluate a batch of override justification records for consistency in how override reasons are applied across similar decision types.

## INPUT DATA
[OVERRIDE_RECORDS_JSON]

## POLICY CONTEXT
[POLICY_DOCUMENT_TEXT]

## INSTRUCTIONS
1. Group the override records by [DECISION_TYPE_FIELD] and [OVERRIDE_REASON_FIELD].
2. For each group, assess whether the override reasons are applied consistently. A consistent group means that similar fact patterns receive the same override reason category and the justifications reference comparable policy grounds.
3. Identify legitimate contextual differences that explain variance (e.g., different risk levels, customer segments, or regulatory requirements explicitly cited in the policy).
4. Flag unexplained variance where similar decisions received different override reasons without a clear contextual justification.
5. Calculate an overall consistency score from 0.0 to 1.0, where 1.0 means all variance is explained by legitimate context and 0.0 means no consistent pattern exists.

## OUTPUT SCHEMA
Return a JSON object with this exact structure:
{
  "consistency_score": number,
  "total_records": number,
  "groups_analyzed": number,
  "consistent_groups": number,
  "divergent_groups": [
    {
      "group_key": string,
      "record_ids": [string],
      "variance_type": "unexplained" | "contextual",
      "explanation": string,
      "recommended_action": string
    }
  ],
  "summary": string
}

## CONSTRAINTS
- Do not invent policy rules not present in [POLICY_DOCUMENT_TEXT].
- If the policy document is insufficient to judge consistency, set consistency_score to null and explain why in the summary.
- Flag but do not resolve ambiguous cases; note them for human review.
- If fewer than 5 records are provided, note that statistical confidence is low.

To adapt this template, replace [OVERRIDE_RECORDS_JSON] with your batch of override records serialized as a JSON array. Each record should contain at minimum a unique ID, the decision type, the override reason, the reviewer's justification text, and any relevant metadata such as risk level or customer segment. Replace [POLICY_DOCUMENT_TEXT] with the full text of your organization's override policy, decision framework, or compliance guidelines. The [DECISION_TYPE_FIELD] and [OVERRIDE_REASON_FIELD] placeholders should be replaced with the actual field names from your records that correspond to the type of decision being overridden and the categorized reason for the override. If your records use different field names, adjust the instructions accordingly.

Before deploying this prompt, validate that your input data is complete and that the policy document is current. In high-stakes governance workflows, route consistency scores below 0.7 and all unexplained divergence groups to a human auditor for review. Log every scoring run with the input batch hash, policy version, and model response for auditability. For production systems, consider running this prompt on a scheduled basis against recent override batches and alerting on score degradation trends.

This prompt is not a one-shot analysis tool; it is a scoring engine that must be embedded in a governance workflow. The harness receives a batch of override records—each containing a decision type, a justification, and reviewer metadata—and returns a consistency score with divergence examples. The primary integration point is a scheduled audit job or an on-demand review triggered by a governance lead. The harness must normalize for legitimate contextual differences (e.g., different risk levels for the same decision type) while flagging unexplained variance. This means the application layer must pre-process override records to extract decision type, risk tier, and any policy-relevant context before calling the prompt.

The implementation should follow a validate → score → review loop. First, validate that each override record in the batch contains the required fields: decision_type, justification_text, reviewer_id, and risk_tier. Reject or quarantine records with missing fields before scoring. Second, call the prompt with a batch size that fits the model's context window—start with 20-30 records per call and paginate larger batches. The prompt returns a JSON object with consistency_score (0.0 to 1.0), divergence_examples (an array of record pairs with conflicting justifications), and normalization_notes explaining which differences were treated as legitimate context. Third, route results to a human reviewer when the consistency score falls below a defined threshold (e.g., 0.7) or when divergence examples indicate potential policy drift. Log every scoring run with the batch ID, timestamp, model version, and reviewer disposition for auditability.

Model choice and retries: Use a model with strong reasoning and structured output capabilities (e.g., GPT-4o, Claude 3.5 Sonnet). Set temperature=0 for deterministic scoring. Implement a retry layer that catches JSON parse failures and schema validation errors—if the output does not match the expected schema, retry once with the validation error message appended to the prompt. If the retry also fails, log the failure and escalate the batch for manual review. Evaluation and calibration: Before production use, run the prompt against a golden dataset of override batches with known consistency issues. Measure agreement between the prompt's divergence examples and human-annotated inconsistencies. Tune the consistency score threshold based on your organization's risk tolerance. What to avoid: Do not use this prompt for real-time override decisions—it is a batch analysis tool. Do not treat the consistency score as a performance metric for individual reviewers without human review of divergence examples. Always pair automated scoring with a governance lead's judgment before taking corrective action.