Human Review Routing Prompt Based on Confidence

This prompt is designed for operations teams who have already completed the extraction and confidence-scoring phase of a document processing pipeline. You have a structured payload containing extracted fields, each annotated with a confidence score, null reason codes, and source spans. The job-to-be-done is not to improve the extraction itself, but to make a deterministic, auditable routing decision about which records and individual fields require human review before the data can be trusted in downstream systems. The ideal user is a pipeline architect or operations engineer who needs to replace ad-hoc, manual sampling with a configurable, rules-based review queue that integrates directly into an application.

Use this prompt when you need to apply configurable routing rules—such as per-field confidence thresholds, record-level aggregate scores, and critical-field overrides—to a batch of extraction results. The prompt expects a well-defined input schema, including the extraction payload and a routing configuration that specifies thresholds, priority weights, and SLI tracking parameters. It produces a structured review queue assignment with priority scores, routing labels, and hooks for tracking service level indicators (SLIs) like time-to-review. This is a deterministic routing layer, not a probabilistic one; the prompt's value is in consistently applying your business rules, not in making subjective judgments about data quality.

Do not use this prompt for the initial extraction step or for generating confidence scores. It assumes that extraction and confidence annotation are already complete and that the input payload conforms to a known schema. If your pipeline lacks per-field confidence scores, or if you need to disambiguate null origins (e.g., missing vs. redacted), you should first apply a prompt like the Confidence-Annotated Field Extraction Prompt Template or the Null Origin Classification Prompt Template. This prompt is also not a replacement for a full human-in-the-loop orchestration system; it handles the routing decision, but you will need to build the surrounding application logic for queue management, reviewer assignment, and SLI monitoring. The next step after reading this section is to examine the prompt template and implementation harness to understand how to wire this decision point into your pipeline.

This prompt template is designed to be pasted directly into your orchestration layer. It accepts an extraction payload containing fields, confidence scores, null reason codes, and source spans, then applies configurable routing rules to determine whether each record should be auto-accepted, escalated for human review, or rejected. Replace every square-bracket placeholder with your specific extraction schema, routing thresholds, queue names, and SLI tracking requirements before deployment.

textCopy
You are a human-review routing engine for an extraction pipeline. Your job is to evaluate extraction records against routing rules and produce a structured review queue assignment.

## INPUT
You will receive an extraction payload in the following JSON schema:
[EXTRACTION_PAYLOAD_SCHEMA]

Example payload:
[EXAMPLE_PAYLOAD]

## ROUTING RULES
Apply the following rules in order. The first matching rule determines the routing decision.

1. **Reject**: If any field in [CRITICAL_FIELD_LIST] has confidence below [CRITICAL_THRESHOLD] OR null_reason is [UNACCEPTABLE_NULL_REASONS], route to REJECT queue with priority = 1.
2. **Escalate**: If any field has confidence below [FIELD_THRESHOLD_MAP] for its field type, OR if ambiguity_flags contains [ESCALATION_FLAGS], route to [REVIEW_QUEUE_NAME] with priority = [PRIORITY_CALCULATION_RULE].
3. **Escalate**: If record_confidence is below [RECORD_THRESHOLD], route to [REVIEW_QUEUE_NAME] with priority = [PRIORITY_CALCULATION_RULE].
4. **Auto-Accept**: Otherwise, route to [ACCEPTED_QUEUE_NAME] with priority = 0.

## OUTPUT SCHEMA
Return a valid JSON object with this exact structure:
{
  "record_id": "string",
  "routing_decision": "ACCEPT" | "ESCALATE" | "REJECT",
  "queue_name": "string",
  "priority": number,
  "matched_rule": "string",
  "triggered_fields": [
    {
      "field_path": "string",
      "confidence": number,
      "threshold": number,
      "reason": "string"
    }
  ],
  "sli_metadata": {
    "routing_timestamp": "ISO8601",
    "total_fields_evaluated": number,
    "fields_below_threshold": number,
    "record_confidence": number
  },
  "human_review_context": {
    "summary": "One-sentence summary of why this record needs review",
    "source_spans": ["span_reference"],
    "ambiguity_notes": ["string"]
  }
}

## CONSTRAINTS
- Never auto-accept a record where [CRITICAL_FIELD_LIST] fields are missing or below threshold.
- Priority must be an integer from 0 (lowest) to [MAX_PRIORITY] (highest).
- If multiple fields trigger escalation, include all of them in triggered_fields.
- If no fields trigger escalation but record_confidence is below threshold, set triggered_fields to an empty array and note record-level confidence as the reason.
- The human_review_context.summary must cite specific field names and confidence values.
- Do not hallucinate fields or confidence values not present in the input.

## EXAMPLES
[FEW_SHOT_EXAMPLES]

## CURRENT PAYLOAD
[EXTRACTION_PAYLOAD]

After pasting this template, replace the placeholders with your production values. [CRITICAL_FIELD_LIST] should include fields whose absence or low confidence makes the entire record unusable—typically identifiers, dates, or regulated data points. [FIELD_THRESHOLD_MAP] defines per-field confidence floors; set these based on calibration benchmarks from your confidence scoring system prompt. [PRIORITY_CALCULATION_RULE] can be a simple formula like 10 - (record_confidence * 10) or a lookup table. The sli_metadata block is essential for tracking routing accuracy over time—pipe it into your observability stack alongside extraction confidence distributions. Validate the output against the schema before enqueueing, and log any records that fail schema validation for prompt debugging.

The routing prompt is not a standalone decision-maker; it is a classification step inside a larger extraction pipeline. The application layer should call this prompt only after field-level extraction and confidence scoring are complete. The prompt expects a structured payload containing per-field confidence scores, null reason codes, and source spans. It returns a routing decision with a priority score and a queue assignment. The application must validate that the input payload is complete before invoking the model, because missing confidence data will produce unreliable routing decisions.

Wire the prompt into a post-extraction processing step. After the primary extraction model returns its annotated JSON, run a lightweight validation check: confirm every required field has a confidence score between 0.0 and 1.0, verify that null reason codes are present where values are absent, and check that source spans are non-empty for extracted values. If validation fails, log the malformed record and route it to a dead-letter queue for manual inspection—do not pass it to the routing prompt. For validated records, assemble the routing prompt with the extraction payload inserted into the [EXTRACTION_OUTPUT] placeholder and the configured thresholds in [CONFIDENCE_THRESHOLDS]. Use a fast, cost-effective model for this classification step (e.g., GPT-4o-mini, Claude Haiku) since the task is structured judgment, not generation. Set temperature=0 to ensure deterministic routing decisions. On the response, validate that the output contains routing_decision, priority_score, and queue_assignment fields. If the model returns an invalid structure, retry once with a stricter schema instruction; if it fails again, escalate the record to a human-operated review queue with a ROUTING_ERROR flag.

Log every routing decision with the record ID, timestamp, model version, confidence scores, priority score, and assigned queue. This audit trail is essential for SLI tracking: measure the rate of auto-approved records, the escalation rate, the human overturn rate on escalated records, and the routing error rate. Set up a periodic eval that samples routed records and checks whether the routing decision aligns with a human reviewer's judgment; use this to tune [CONFIDENCE_THRESHOLDS] and detect threshold drift. Avoid routing high-risk fields (e.g., financial amounts, clinical findings, legal obligations) without a mandatory human review step regardless of confidence score—encode this as a hard rule in the application layer, not just in the prompt. When a record is routed for human review, attach the full extraction payload, source spans, and confidence justifications so the reviewer can make an informed decision without re-reading the entire source document.