Extraction Confidence Tier Assignment Prompt Template

Use this prompt when you have already completed a raw extraction step and need a structured, auditable confidence signal for each field before it lands in a downstream system. The primary job-to-be-done is risk-controlled automation: you want to route high-confidence records directly to your primary database or event stream, queue medium-confidence records for statistical sampling and offline analysis, and send low-confidence or unreliable records to a human review interface. This prompt is designed for data platform engineers, integration developers, and ML ops teams who are building ingestion pipelines where silent extraction errors are more expensive than a human-in-the-loop delay.

The ideal input context includes the raw extracted field value, the specific source text span from which it was extracted, and any available extraction metadata such as the model's internal logprob or a heuristic score. You should not use this prompt as a substitute for a proper extraction model. It does not perform extraction; it evaluates the output of an extraction step. It is also not a replacement for a deterministic validation layer—if a field can be validated against a system of record (e.g., checking an extracted customer ID against a database), do that first and use this prompt for the remaining unstructured fields that lack a ground-truth source. Avoid using this prompt for real-time, sub-millisecond decisioning where the added latency of an LLM call is unacceptable.

Before wiring this into production, define clear action thresholds for each tier. For example, 'high' might mean confidence_score >= 0.9 with a fully grounded source span, while 'unreliable' might mean the source text is missing or contradictory. Map these tiers to your specific infrastructure: a Kafka topic for high-confidence records, an S3 bucket for medium-confidence samples, and a task queue for human review. The next step is to pair this prompt with an evaluation harness that measures tier assignment accuracy against a golden set of human-annotated confidence labels to prevent drift in your automation risk profile.

This template is the core instruction set for a model that must evaluate extraction quality. It forces the model to justify its confidence tier for each field by referencing the source text, not by guessing. Copy the block below into your prompt management system, replace the square-bracket placeholders with your specific extraction schema and source material, and run it against your extracted payloads before they enter your database or review queue.

textCopy
You are an extraction auditor. Your task is to review a set of fields extracted from a source text and assign a confidence tier to each field.

## INPUT
- Source Text:
[SOURCE_TEXT]
- Extracted Fields (JSON):
[EXTRACTED_PAYLOAD]

## CONFIDENCE TIERS
Assign exactly one tier per field:
- **high**: The field value is explicitly stated in the source text with a clear, unambiguous span.
- **medium**: The field value is reasonably inferred from the source text, but the wording is indirect or the value required minor normalization.
- **low**: The field value is a weak guess based on limited or vague context. The source text hints at the value but does not confirm it.
- **unreliable**: The field value appears fabricated, contradicts the source text, or has no detectable basis in the source text. Also use this tier if the field is missing from the source entirely but the extraction produced a value.

## CONSTRAINTS
[CONSTRAINTS]

## OUTPUT SCHEMA
Return a valid JSON object with this structure:
{
  "fields": [
    {
      "field_name": "string",
      "confidence_tier": "high|medium|low|unreliable",
      "evidence": "Quote the exact source span that supports this tier, or explain why no span exists.",
      "needs_human_review": true|false
    }
  ],
  "overall_assessment": "Brief summary of extraction quality and any systemic issues noticed."
}

## RULES
1. Do not change the extracted field values. Only assess them.
2. If a field is missing from the extracted payload, do not include it in your output.
3. Set `needs_human_review` to true for any field with a `low` or `unreliable` tier.
4. If the source text is empty or nonsensical, mark all fields as `unreliable` and explain why in the overall assessment.
5. For [REGULATED_DOMAIN] use cases, always set `needs_human_review` to true for any field that could trigger a compliance obligation, regardless of confidence tier.

To adapt this template, replace [SOURCE_TEXT] with the raw document or snippet the extraction ran against. Replace [EXTRACTED_PAYLOAD] with the JSON output from your extraction step. Use [CONSTRAINTS] to inject domain-specific rules, such as 'Dates must be in ISO 8601 format' or 'Currency values must include the ISO 4217 code.' If you are operating in a regulated domain like healthcare, finance, or legal, replace [REGULATED_DOMAIN] with the specific domain name to activate the mandatory human-review rule for compliance-triggering fields.

Before deploying, test this prompt against a golden set of extractions where you already know the correct confidence tiers. Compare the model's tier assignments and evidence quotes to your ground truth. Pay special attention to fields the model marks as high but that contain hallucinated values—this is the most dangerous failure mode. If you see tier inflation, tighten the tier definitions in the prompt or add few-shot examples of correctly tiered borderline cases. Wire the needs_human_review boolean into your downstream routing logic so that low-confidence records never land in an automated database without inspection.

The confidence tier assignment prompt is not a standalone classifier—it is a pipeline gate. Its output determines whether extracted fields proceed to downstream systems, enter a human review queue, or are discarded. The implementation harness must treat this prompt as a deterministic decision point with clear contracts: input is a set of extracted fields with their source spans, output is a tier assignment per field plus a routing recommendation. Wire this prompt after extraction and before ingestion, so no unverified field reaches your database or API.

Validation layer: Parse the model's JSON output and validate it against a strict schema before acting on it. Check that every input field receives exactly one tier (high, medium, low, unreliable), that the routing field contains a valid enum (auto_ingest, review_queue, reject), and that any evidence_span references point to real positions in the source text. If validation fails, retry once with the validation errors injected into the prompt as [PREVIOUS_ERRORS]. After two failures, route the entire record to human review and log the failure for prompt debugging. Model selection: Use a model with strong instruction-following and JSON mode support (GPT-4o, Claude 3.5 Sonnet, or equivalent). Avoid smaller models for this task—confidence calibration degrades sharply below the ~70B parameter threshold, and you need consistent tier boundaries across thousands of records.

Retry and fallback logic: Implement a two-strike retry policy. On first validation failure, append the specific schema violations to a [CORRECTION_HINT] placeholder and re-invoke. On second failure, escalate. Logging: Capture the input fields, assigned tiers, model used, latency, and any validation errors for every invocation. This log becomes your audit trail when downstream consumers question why a field was ingested or rejected. Human review integration: When the routing recommendation is review_queue or after retry exhaustion, push the record to a review interface that displays the source text, extracted fields, model-assigned tiers, and evidence spans side by side. Allow reviewers to override tiers and provide a reason code. Feed those overrides back into your eval dataset for regression testing.

What to avoid: Do not use this prompt as a real-time synchronous call in a user-facing flow—confidence tier assignment is a batch or async pipeline stage. Do not skip validation and trust the model's output shape; schema drift is the most common production failure. Do not route low or unreliable fields to automated ingestion without a human-in-the-loop override mechanism. And do not treat the confidence tier as a calibrated probability—it is a relative ordering signal, not a statistical confidence interval. For high-stakes domains (healthcare, legal, finance), require human review for all medium and below tiers, not just low and unreliable.