Escalation Justification Extraction Prompt

This prompt is designed for compliance officers, audit reviewers, and AI operations engineers who need to systematically extract and classify the stated reason for every escalation decision from production trace metadata. The core job-to-be-done is transforming unstructured or semi-structured trace logs into an audit-ready justification log, where each escalation event is paired with a classified reason, a completeness score, and flags for missing or generic justifications. You should use this prompt when you have a batch of trace events that include an escalation step—such as a handoff from one model tier to another, a fallback activation, or a human-in-the-loop transfer—and you need to prove to an internal or external auditor that each escalation was justified, documented, and compliant with policy.

The ideal user is someone who already has access to structured trace metadata (e.g., JSON log entries, span attributes, or routing decision payloads) and needs to convert that raw data into a human-readable, defensible record. The prompt expects an [INPUT] that contains one or more escalation trace objects, each with fields like escalation_timestamp, source_model, target_model, reason_code, reason_text, and any surrounding context. It also accepts an optional [TAXONOMY] of acceptable justification types (e.g., LATENCY_EXCEEDED, QUALITY_BELOW_THRESHOLD, CAPABILITY_GAP, ERROR_CODE, POLICY_MANDATED, UNKNOWN) and a [COMPLETENESS_RUBRIC] that defines what makes a justification 'complete' versus 'generic' or 'missing.' Do not use this prompt for real-time routing decisions, for generating justifications where none exist, or for making normative judgments about whether an escalation was correct—only for extracting and classifying what was recorded.

Before running this prompt, ensure your trace data includes the escalation reason field. If your system logs escalations without a reason, this prompt will flag them as MISSING and score them zero for completeness, which is itself a valuable audit finding. The output is a structured justification log—typically a JSON array—where each record includes the original escalation identifier, the extracted justification text, the classified type, a completeness score, and any flags. The next step after extraction is to feed this log into your governance review process, where human reviewers can investigate flagged items and confirm that the documented reasons align with the actual trace context. Avoid using this prompt on traces that have already been summarized or redacted, as the extraction accuracy depends on the raw reason field and its surrounding metadata.

This prompt is designed to be invoked programmatically against individual trace records or small batches. It expects structured trace metadata as input and produces a normalized justification log entry. The template uses square-bracket placeholders for all variable inputs—replace these with actual values from your trace store before sending the request to the model. For high-compliance environments, always route the output through a human review step before committing to an audit record.

textCopy
You are an audit assistant reviewing escalation decisions from production AI traces.

Your task is to extract the stated reason for each escalation, classify it, and assess its completeness.

## INPUT
Trace metadata for one or more escalation events:
[TRACE_METADATA]

## OUTPUT_SCHEMA
Return a JSON array of justification records. Each record must follow this schema:
{
  "trace_id": "string",
  "escalation_step": "integer",
  "stated_reason": "string or null",
  "reason_classification": "capability_gap | quality_threshold | timeout | error_code | cost_limit | policy_rule | manual_override | unknown",
  "completeness_score": "float between 0.0 and 1.0",
  "completeness_flags": ["string"],
  "source_field": "string"
}

## CLASSIFICATION RULES
- capability_gap: The model lacked a required capability or tool.
- quality_threshold: Output quality, confidence, or eval score fell below a configured threshold.
- timeout: The primary model exceeded a latency deadline.
- error_code: An explicit error code or exception was returned.
- cost_limit: The request would have exceeded a cost or token budget.
- policy_rule: A governance, safety, or compliance policy triggered escalation.
- manual_override: A human or external system forced the escalation.
- unknown: No discernible reason could be extracted.

## COMPLETENESS RULES
Score 1.0 if the stated reason is specific, references a measurable condition, and includes a threshold or error identifier.
Score 0.5 if the reason is present but generic (e.g., "fallback activated" or "quality issue").
Score 0.0 if no reason is stated or the field is empty.

Flag any of the following if detected:
- "missing_reason": No stated reason found.
- "generic_language": Reason uses vague or non-actionable language.
- "no_threshold": Reason mentions a condition but no specific threshold.
- "conflicting_signals": Multiple reasons present that contradict each other.
- "truncated": The reason field appears cut off or incomplete.

## CONSTRAINTS
- Do not fabricate reasons. If the trace contains no justification, set stated_reason to null and classify as "unknown".
- Do not guess classifications. Use only the evidence present in the trace metadata.
- Preserve the original reason text verbatim in stated_reason—do not paraphrase.
- If multiple escalation steps exist in one trace, produce one record per step.

## EXAMPLES
Input: {"trace_id": "t-001", "escalation_step": 1, "metadata": {"fallback_reason": "primary_model_timeout_ms=5000"}}
Output: {"trace_id": "t-001", "escalation_step": 1, "stated_reason": "primary_model_timeout_ms=5000", "reason_classification": "timeout", "completeness_score": 1.0, "completeness_flags": [], "source_field": "metadata.fallback_reason"}

Input: {"trace_id": "t-002", "escalation_step": 1, "metadata": {"fallback_reason": "quality degraded"}}
Output: {"trace_id": "t-002", "escalation_step": 1, "stated_reason": "quality degraded", "reason_classification": "quality_threshold", "completeness_score": 0.5, "completeness_flags": ["generic_language", "no_threshold"], "source_field": "metadata.fallback_reason"}

Input: {"trace_id": "t-003", "escalation_step": 1, "metadata": {}}
Output: {"trace_id": "t-003", "escalation_step": 1, "stated_reason": null, "reason_classification": "unknown", "completeness_score": 0.0, "completeness_flags": ["missing_reason"], "source_field": "none"}

## RISK_LEVEL
[HIGH] — Outputs from this prompt may be used in compliance audits. Human review is required before finalizing any audit record.

To adapt this template, replace [TRACE_METADATA] with the actual JSON payload from your trace store. If your trace schema uses different field names for the escalation reason, update the source_field mapping and the classification logic accordingly. For batch processing, ensure each trace object includes a unique trace_id and escalation_step so outputs remain traceable. If your observability stack already enriches traces with reason codes, add those codes to the classification enum to avoid unnecessary model inference. Always run the output through a schema validator before ingestion—malformed JSON or missing required fields should trigger a retry or fallback to manual review.

This prompt is designed to operate as a batch processing step within a compliance audit workflow, not as a real-time user-facing feature. The primary integration pattern is a scheduled job or event-driven function that pulls closed escalation traces from your observability store, runs each trace through the prompt, and writes structured justification records to an audit database. Because the output feeds compliance reporting, the implementation must treat every extraction as an evidence artifact: the raw prompt input, the model response, and any validation results should be logged immutably with trace IDs and timestamps before any downstream consumption.

Wire the prompt into an application by wrapping it in a processing function that enforces a strict contract. The function should accept a single trace object containing the escalation event metadata, the upstream model response that triggered the escalation, and any configured escalation policy text as [POLICY_CONTEXT]. Before calling the model, validate that required fields are present and non-empty—return a structured SKIPPED record with a reason code if the trace is incomplete. After receiving the model response, parse the JSON output and run a schema validator that checks for required fields (justification_text, justification_type, completeness_score), enum membership for justification_type (e.g., capability_gap, timeout, quality_threshold, cost_exceeded, policy_rule, unknown), and numeric range for completeness_score (0.0–1.0). If validation fails, retry once with the same input and an added [REPAIR_INSTRUCTION] that includes the specific validation error. If the retry also fails, route the trace to a human review queue with the raw model output and validation errors attached.

Model choice matters here. Use a model with strong structured output compliance and low refusal rates on audit tasks—GPT-4o or Claude 3.5 Sonnet are appropriate defaults. Avoid smaller or older models that may hallucinate justification types or produce malformed JSON under batch load. Set temperature=0 to maximize consistency across traces, and configure a request timeout of 30 seconds with a circuit breaker that halts processing if the model endpoint returns 5xx errors for more than 10% of requests in a rolling window. For high-volume pipelines, implement a dead-letter queue that captures traces that fail after retries so they can be replayed once the upstream issue is resolved.

Human review is required when the prompt flags a justification as missing or generic, or when the completeness_score falls below a configurable threshold (start with 0.4 and calibrate based on audit sampling). Route these cases to a review interface that displays the original trace metadata, the model's extracted justification, and a simple form for the reviewer to confirm, correct, or reclassify the justification. The reviewer's decision should be logged as the authoritative record and can be used to build a fine-tuning dataset if the prompt consistently misclassifies certain escalation patterns.

For observability, instrument the processing function with metrics: extraction latency per trace, validation pass/fail rate, retry rate, human-review escalation rate, and justification type distribution over time. These metrics will surface prompt drift—for example, a sudden increase in unknown justification types may indicate that new escalation paths were added to the router without corresponding updates to the prompt's classification taxonomy. Log every model call with the full prompt, response, and validation result to enable trace-level debugging when audit reviewers question a specific justification record.