Human Review Queue Triage Audit Prompt

This prompt is for operations and data platform engineers who need to validate the automated routing decisions in their extraction pipeline. The core job-to-be-done is to close the feedback loop between a model's confidence score and the actual quality of its output. You run this audit when you have a batch of records, each containing the raw extraction output, the model's self-assigned confidence score, and the binary triage decision that routed the record to either automated ingestion or a human review queue. The goal is not to re-extract data, but to judge whether the triage logic made the correct call.

The ideal user has direct access to the extraction pipeline's output store and the triage router's decision log. You need three data points per record to make this work: the extracted fields, the confidence score that triggered the routing decision, and the source text the extraction was performed on. Without the source text, the audit cannot verify extraction quality and becomes a simple statistics report, which misses the point. This prompt is designed for batch operations, run weekly or per ingestion window, before downstream systems consume the extracted data. It is not a real-time guard; it is a retrospective quality control step.

Do not use this prompt when you lack the source documents, when you need a real-time triage decision, or when your pipeline does not yet produce confidence scores. It is also the wrong tool if you are trying to debug a single bad extraction—use the Extraction Error Categorization Prompt for that. Avoid running this audit on streaming data without batching; the prompt is designed to reason over a set of records to identify systemic patterns, not individual anomalies. After running this audit, expect to update your confidence threshold, refine your routing rules, or investigate specific failure modes flagged in the output.

This prompt template is designed to be dropped into an audit workflow that compares triage decisions (escalated vs. not escalated) against the actual quality of extracted records. It expects a batch of records, each containing the original triage decision, the extraction output, and the source text. The model acts as an auditor, not a triage agent—it judges whether the original routing was correct, identifies misclassifications, and produces a structured audit report suitable for operations review and queue performance measurement.

textCopy
You are an extraction quality auditor. Your task is to review a batch of records that were routed through a human review triage system. Each record contains the original triage decision, the extracted fields, and the source text. You must determine whether the triage decision was correct and flag any misclassifications.

## INPUT
You will receive a JSON array of triage records. Each record has this structure:
{
  "record_id": "string",
  "triage_decision": "escalated" | "not_escalated",
  "triage_reason": "string (original reason for the decision)",
  "extracted_fields": { ... },
  "extraction_confidence": 0.0-1.0,
  "source_text": "string (the original document text)",
  "ground_truth_available": true | false,
  "ground_truth_fields": { ... } | null
}

## AUDIT CRITERIA
For each record, evaluate the triage decision against these criteria:
1. **Should have been escalated but wasn't (missed escalation):** The record was marked "not_escalated" but contains extraction errors, missing required fields, confidence below [CONFIDENCE_THRESHOLD], or values that contradict the source text.
2. **Was escalated unnecessarily (false escalation):** The record was marked "escalated" but the extraction is complete, accurate, and confidence meets or exceeds [CONFIDENCE_THRESHOLD].
3. **Correct escalation:** The record was marked "escalated" and the extraction genuinely has quality issues warranting review.
4. **Correct non-escalation:** The record was marked "not_escalated" and the extraction is complete and accurate.

## CONSTRAINTS
[CONSTRAINTS]

## OUTPUT SCHEMA
Return a JSON object with this exact structure:
{
  "audit_summary": {
    "total_records": number,
    "correct_decisions": number,
    "missed_escalations": number,
    "false_escalations": number,
    "accuracy_rate": number (0.0-1.0),
    "review_queue_waste_percentage": number (false escalations / total escalated)
  },
  "findings": [
    {
      "record_id": "string",
      "triage_decision": "escalated" | "not_escalated",
      "audit_judgment": "correct_escalation" | "correct_non_escalation" | "missed_escalation" | "false_escalation",
      "severity": "critical" | "high" | "medium" | "low",
      "evidence": ["specific issues found, referencing source text spans"],
      "recommended_action": "string (e.g., 'route to review', 'clear from queue', 're-extract')"
    }
  ],
  "patterns_identified": ["string observations about systematic issues"],
  "queue_health_assessment": "string (overall assessment of triage quality)"
}

## SEVERITY GUIDELINES
- **critical:** Missed escalation where extracted values are hallucinated or contradict source text; could cause downstream harm if consumed.
- **high:** Missed escalation with missing required fields or confidence below threshold by more than 0.2.
- **medium:** False escalation wasting reviewer time; minor extraction issues that don't affect core fields.
- **low:** Correct decisions with minor edge-case notes.

## EXAMPLES
[EXAMPLES]

## RISK LEVEL
[RISK_LEVEL]

Audit the following batch:
[INPUT]

To adapt this template, replace the square-bracket placeholders with your specific parameters. Set [CONFIDENCE_THRESHOLD] to your pipeline's escalation threshold (commonly 0.7–0.85). Populate [CONSTRAINTS] with domain-specific rules such as required field lists, regulatory requirements, or PII handling policies. Provide [EXAMPLES] as few-shot demonstrations showing correct audit judgments for your domain's typical failure patterns. Set [RISK_LEVEL] to guide the auditor's sensitivity—use "high" for regulated domains where missed escalations are unacceptable, or "moderate" for internal analytics where false escalations are more tolerable than missed ones. The [INPUT] placeholder should receive your batch of triage records as a JSON array. After running the audit, always route findings with severity "critical" or "high" for human confirmation before acting on the audit results. For production pipelines, validate the output JSON against the schema before ingesting findings into your operations dashboard or review queue management system.

This prompt is designed to run as a batch audit step, not a real-time triage gate. It should be executed periodically (e.g., hourly, daily) against a sample or full population of recently triaged items. The harness pulls records from the review queue's backing store—including the original extraction payload, the triage decision (escalated or auto-processed), and any confidence scores—and feeds them into the prompt for retrospective judgment. The output is a structured audit report that flags false negatives (items that should have been escalated but weren't) and false positives (items escalated unnecessarily), enabling the operations team to tune triage thresholds and retrain routing rules.

Wire the prompt into an application using a three-stage pipeline: (1) a fetcher that queries the review queue database for triaged items within the audit window, joining extraction records with their triage outcomes; (2) a batch processor that formats each item into the [AUDIT_BATCH] placeholder—typically a JSON array of objects containing the source text, extracted fields, confidence scores, triage decision, and any reviewer notes—and calls the LLM with the prompt template; (3) a validator that parses the structured output against the expected [OUTPUT_SCHEMA] and writes results to an audit log table. Use structured output mode (e.g., JSON mode or function calling with a strict schema) to enforce the audit report format. If the model returns malformed JSON or missing required fields, retry once with a repair prompt that includes the validation error. After two failures, log the raw response and flag for human review.

For model choice, prefer a model with strong instruction-following and structured output support (e.g., GPT-4o, Claude 3.5 Sonnet) because the audit requires precise comparison between extraction quality and triage decisions. Avoid smaller or older models that may conflate confidence scores or hallucinate audit findings. Implement logging that captures the prompt version, model version, batch ID, timestamp, and full audit output for traceability. Store audit results in a table with columns for item_id, triage_decision, audit_judgment (correct_escalation, missed_escalation, unnecessary_escalation, correct_auto_process), audit_reason, and audit_timestamp. This structure enables downstream dashboards that track triage accuracy over time. Do not use this prompt to automatically change triage decisions—it is an audit tool, not a corrective action. Any systemic issues it surfaces should trigger a human review of triage thresholds or prompt improvements before changes are deployed.