Escalation Criteria Definition Prompt Template

This prompt is designed for verification system architects and platform engineers who need to move from informal, tribal-knowledge escalation rules to a structured, testable, and machine-readable criteria specification. The core job-to-be-done is producing a document that a routing engine can execute: a set of conditions, thresholds, and logical rules that determine whether a claim proceeds to auto-verification, gets flagged for human review, or is escalated as high-risk. The ideal user is someone who already has production logs, example claims, and a clear understanding of their operational cost and latency budgets. They are not starting from scratch; they are formalizing existing intuition into a specification that can be version-controlled, tested against historical data, and deployed as part of a verification pipeline.

You should use this prompt when you have concrete examples of both routine and edge-case claims, and you need to eliminate ambiguity in your triage logic. The prompt requires several structured inputs to be effective: a set of [CLAIM_EXAMPLES] that represent your production distribution, your known [FAILURE_MODES] (e.g., false negatives on numerical claims, over-escalation of domain-specific jargon), your [OPERATIONAL_CONSTRAINTS] (e.g., human review capacity, latency budget, cost per auto-verification), and a target [OUTPUT_SCHEMA] for the criteria. Do not use this prompt if you lack production data to validate the output; generating criteria in a vacuum will produce a specification that looks plausible but fails on real traffic. This prompt is also not a substitute for a routing implementation—it defines the what, not the how of execution.

The primary constraint is that the generated criteria must be executable and non-contradictory. A common failure mode is producing rules that overlap in ways that create dead zones (claims that match no rule) or conflict zones (claims that match multiple rules with different outcomes). The harness must test for these conditions by running the criteria against a golden dataset of claims with known correct escalation outcomes. Additionally, the criteria must handle edge cases explicitly: claims with missing evidence, claims in domains where your knowledge base is weak, and claims that are syntactically simple but semantically high-risk. The next step after generating the criteria is to run them through a regression test suite that includes adversarial examples designed to probe rule boundaries, and to log any criteria violations found in production for iterative refinement.

This prompt template is designed for verification system architects who need to define and document the exact rules for when a claim should be escalated for human review. It takes your system's constraints, risk tolerances, and historical data as input and produces a structured specification that can be directly translated into application logic. The output is not just a list of rules; it includes conditions, thresholds, illustrative examples, and edge cases to ensure the criteria are executable and non-contradictory.

textCopy
You are an expert verification system architect. Your task is to produce a structured, executable escalation criteria specification based on the provided inputs. The specification must be unambiguous, testable, and cover known edge cases.

## INPUTS
- System Requirements: [SYSTEM_REQUIREMENTS]
- Risk Tolerance Policy: [RISK_TOLERANCE_POLICY]
- Production Logs (Anonymized Escalation Scenarios): [PRODUCTION_LOGS]
- Domain Context: [DOMAIN_CONTEXT]

## OUTPUT SCHEMA
Generate a JSON object with the following structure:
{
  "escalation_rules": [
    {
      "rule_id": "string",
      "rule_name": "string",
      "condition": "string (a precise, testable logical expression)",
      "thresholds": {
        "metric_name": "value"
      },
      "escalation_reason": "string (explains the risk or policy violated)",
      "priority": "string (CRITICAL | HIGH | MEDIUM | LOW)",
      "example_scenario": "string (a concrete example from the logs that triggers this rule)",
      "edge_case_handling": "string (how to handle borderline or ambiguous inputs)"
    }
  ],
  "global_thresholds": {
    "auto_verify_confidence_min": "number",
    "human_review_confidence_max": "number",
    "max_auto_verification_latency_ms": "number"
  },
  "contradiction_checks": [
    "string (a statement describing a potential conflict between rules and how it is resolved)"
  ],
  "unresolved_scenarios": [
    "string (a scenario from the logs that cannot be fully resolved by the current rules and requires further definition)"
  ]
}

## CONSTRAINTS
- Every rule must be directly traceable to a system requirement or a pattern in the production logs.
- Conditions must be written as testable logical expressions (e.g., 'claim.confidence_score < 0.8 AND claim.domain == "medical"').
- The contradiction_checks array must explicitly state how conflicting rules are prioritized.
- If a scenario from the logs cannot be covered, it must be listed in unresolved_scenarios with a reason.
- Do not invent rules that are not supported by the provided inputs.

To adapt this template, replace the square-bracket placeholders with your specific context. The [SYSTEM_REQUIREMENTS] should be a clear list of your verification pipeline's goals and non-goals. [PRODUCTION_LOGS] is critical; provide a representative sample of anonymized cases where the system succeeded, failed, or was uncertain. After generating the specification, the next step is to run it through a validation harness that tests each rule against the provided log scenarios to ensure the conditions are executable and the rules do not contradict each other. For high-stakes domains like healthcare or finance, a human expert must review the final specification before it is implemented in code.

This prompt is not a one-off document generator; it is a specification compiler that must produce machine-executable rules. The harness should treat the generated criteria as code. After the model returns a structured criteria specification, the first integration step is a syntactic validation pass: parse the output schema, confirm that every condition has a defined threshold, verify that all escalation paths terminate in a concrete action (auto-verify, human review, or reject), and check for contradictory rules. A common failure mode is criteria that look plausible but contain unreachable branches—for example, a rule that escalates claims with confidence below 0.6 to human review while another rule auto-verifies all claims with confidence above 0.5, leaving no behavior for the 0.5–0.6 band. The harness must flag these gaps before the criteria enter production.

Testing against production logs is mandatory. Before deploying generated criteria, replay a representative sample of historical verification decisions through the new rules and compare routing outcomes. The harness should measure three signals: recall on known escalations (did the criteria catch claims that previously required human intervention?), false escalation rate (how many claims that were safely auto-verified now route to reviewers?), and coverage of edge cases (do the criteria explicitly handle the top 20 failure patterns from your incident log?). If the criteria miss known high-risk patterns—such as claims involving regulated domains, numerical assertions without source data, or quotes attributed to unnamed sources—reject the output and retry with those failure cases injected into the [EXAMPLES] placeholder. A single pass through the prompt is rarely sufficient; expect to iterate with production-derived counterexamples.

Human review of the criteria themselves is a hard gate, not optional. The generated specification should be surfaced in a review interface that lets verification operations leads inspect each rule, test it against sample claims, and approve or reject individual conditions. The harness must support partial approval: a reviewer might accept domain-specific escalation rules for financial claims but reject overly broad sensitivity triggers that would flood the queue. Log every approval decision with the reviewer identity, timestamp, and rule version. This audit trail is essential for governance reviews and for debugging when escalation behavior changes after a criteria update. Never auto-apply generated criteria without human sign-off—the cost of a misrouted high-risk claim far exceeds the latency of a review cycle.

Model choice and fallback matter here. Use a model with strong reasoning and structured output discipline (GPT-4o, Claude 3.5 Sonnet, or equivalent) for the primary generation. If the primary model produces criteria that fail syntactic validation twice, fall back to a simpler template-based criteria generator with hardcoded safety overrides, then flag for manual authoring. The harness should also version every generated criteria set alongside the prompt template, input parameters, and production log sample used for testing. When escalation behavior drifts—and it will—this version history lets you trace whether the change came from a prompt edit, a model update, or a shift in claim patterns. Wire criteria deployment through the same CI/CD pipeline you use for service configuration changes, with canary routing to a small percentage of verification traffic before full rollout.