Field Extraction Certainty Rubric Prompt

This playbook is for data quality engineers, MLOps teams, and platform architects who need to standardize how confidence scores are assigned across different extraction models, prompts, and human reviewers. The core problem is inconsistency: one extractor's 0.8 is another's 0.5. This prompt defines a 5-level certainty rubric with concrete criteria, examples, and edge cases for each level. Use it when you need to calibrate a new extraction task, audit an existing pipeline, or generate training data for a confidence classifier.

Deploy this prompt before you start scoring extraction outputs at scale. It acts as a meta-policy that other prompts and human reviewers will follow. For example, if you are building a document intelligence pipeline that extracts contract dates, you would first run this prompt to produce a rubric defining what 'High Confidence' means for a date field—perhaps requiring an explicit, unambiguous date string in a standard format. That rubric then becomes the [RUBRIC] input for your downstream extraction prompts. Do not use this prompt for the extraction itself; it is a calibration tool, not an extractor.

Avoid this prompt when you have a single, well-calibrated model with a known confidence distribution that doesn't need cross-system alignment. If your team is not yet experiencing confidence score inconsistency across extractors or reviewers, start with the Confidence-Annotated Field Extraction Prompt Template instead. For teams already in production with drift issues, pair this rubric prompt with the Confidence Score Drift Detection Prompt to monitor when recalibration is needed.

This prompt template encodes a standardized 5-level certainty rubric directly into the extraction instructions. Every extracted field must be assigned a certainty level—Definite, High, Moderate, Low, or Uncertain—based on explicit criteria tied to source evidence quality, ambiguity, and inference chains. The template is designed to be dropped into an extraction pipeline where multiple extractors (human or LLM) need to apply the same confidence standard, making downstream calibration, routing, and audit comparisons possible.

textCopy
You are an extraction specialist operating under a strict 5-level certainty rubric. For every field you extract from the [SOURCE_TEXT], you must assign exactly one certainty level and provide a brief justification citing the specific evidence or lack thereof.

## Certainty Rubric

**Definite (Level 5)**
- Criteria: The value is explicitly stated in the source text with no ambiguity. The exact string match is present. No inference or interpretation is required.
- Example: Source says "Invoice total: $1,250.00" → Extracting invoice_total as 1250.00 is Definite.
- Edge case: If the value appears but uses a different format (e.g., "1,250" vs "1250"), this is still Definite if the numeric value is unambiguous.

**High (Level 4)**
- Criteria: The value is clearly implied by the source text through straightforward, single-step reasoning. Multiple source spans converge on the same value. Minor normalization required but no alternative interpretations are plausible.
- Example: Source says "Payment due 30 days from invoice date of March 1" → Extracting due_date as March 31 is High (single-step date arithmetic).
- Edge case: If the source contains a calculation that could be performed two ways but context makes one interpretation dominant, this is High, not Moderate.

**Moderate (Level 3)**
- Criteria: The value requires multi-step inference, resolution of minor contradictions, or selection among plausible alternatives. The extracted value is the most likely interpretation but other reasonable interpretations exist.
- Example: Source mentions "the client" in paragraph 1 and names "Acme Corp" in paragraph 4 with no other clients mentioned → Extracting client_name as "Acme Corp" is Moderate (coreference resolution across distance).
- Edge case: When two source spans provide conflicting values and one is clearly more authoritative (e.g., a signature block vs body text), the extraction is Moderate, not High.

**Low (Level 2)**
- Criteria: The value is guessed from weak signals, domain defaults, or statistical likelihood. The source text does not directly support the value but does not contradict it either. Multiple plausible values exist with no clear winner.
- Example: Source describes a "software license" without specifying term length → Extracting license_term as "12 months" based on industry default is Low.
- Edge case: If the source is silent on a field but the field is required by the schema, mark it Low with justification "value assumed from domain default; source is silent."

**Uncertain (Level 1)**
- Criteria: The source text is contradictory without resolution, the relevant span is illegible or redacted, or the model cannot determine any value with confidence above random chance. This is the appropriate level when the model should escalate for human review.
- Example: Source says "Contract value: $50,000" in section 2 and "Total consideration: $75,000" in section 5 with no reconciliation → Extracting contract_value is Uncertain.
- Edge case: If the source contains the field name but the value is redacted (e.g., "Amount: [REDACTED]"), mark as Uncertain with justification "value redacted in source."

## Input

Source text to extract from:
[SOURCE_TEXT]

Fields to extract with their descriptions:
[FIELD_SCHEMA]

## Output Format

Return a JSON object with this structure:
{
  "extractions": [
    {
      "field_name": "string",
      "value": <extracted value or null>,
      "certainty_level": "Definite" | "High" | "Moderate" | "Low" | "Uncertain",
      "certainty_score": <integer 1-5 corresponding to level>,
      "justification": "string citing specific source evidence or reason for uncertainty",
      "source_spans": ["exact text from source supporting the extraction"],
      "inference_chain": ["step-by-step reasoning if inference was used, empty array if Definite"]
    }
  ],
  "extraction_metadata": {
    "overall_confidence": <float 0.0-1.0>,
    "fields_requiring_review": ["field_names with certainty_level Low or Uncertain"],
    "rubric_version": "1.0"
  }
}

## Constraints
- Never assign Definite to an inferred value, even if you are highly confident in the inference.
- If the source text is completely silent on a field, mark it Uncertain (Level 1), not Low.
- If two source spans conflict and you cannot resolve the conflict, mark Uncertain and include both spans.
- Justifications must reference specific source text, not general reasoning.
- If [RISK_LEVEL] is "high", escalate all Moderate, Low, and Uncertain fields for human review in extraction_metadata.

Adaptation guidance: Replace [SOURCE_TEXT] with the document or passage being processed. Replace [FIELD_SCHEMA] with an array of field definitions, each containing name, description, type, and required boolean. Set [RISK_LEVEL] to "high" for regulated domains (legal, clinical, financial) to force broader human review; use "standard" for internal analytics where Moderate extractions can flow through. If your application uses a different output schema, adjust the JSON structure but preserve the five certainty levels, justification requirement, and source span anchoring. For batch extraction, wrap this prompt in a loop that processes one document per call to keep justifications grounded in specific source text rather than aggregated across documents.

This prompt is not a standalone extractor—it is a calibration and annotation layer that wraps an existing extraction step. In practice, you run your primary field extraction prompt first, then feed its output (raw extracted values with source spans) into this rubric prompt to assign a standardized 5-level certainty score to each field. The rubric prompt expects the extraction context, the field name, the extracted value, and the source evidence as inputs. It returns a certainty level (1–5), a justification string, and a flag for whether human review is recommended. This separation keeps extraction logic and confidence logic independently testable and tunable.

Wire the prompt into your pipeline as a post-extraction processor. After the primary extractor returns a JSON record, iterate over each extracted field and call this rubric prompt with the field's source text, extracted value, and any surrounding context. Validate the response against a strict output schema: certainty_level must be an integer 1–5, justification must be a non-empty string, and human_review_recommended must be a boolean. If the model returns an invalid level or omits a required field, retry once with a stronger constraint instruction appended. Log every rubric call with the input context, output certainty, and justification for audit trails. For high-risk domains (legal, clinical, financial), route any field scored 1 or 2 directly to a human review queue before the record enters downstream systems.

Model choice matters here. Use a model with strong instruction-following and structured output support (GPT-4o, Claude 3.5 Sonnet, or equivalent). Avoid smaller models that may collapse the 5-level scale into binary high/low judgments. Set temperature=0 for deterministic scoring. If you observe score inflation (most fields clustering at 4–5), calibrate by adding few-shot examples of genuinely ambiguous extractions scored at 2–3. Store rubric outputs alongside extraction results in your data warehouse so you can monitor score distributions over time, detect drift, and tune thresholds. Do not use this rubric prompt to make automated rejection decisions without a human-in-the-loop fallback for low-certainty fields—the rubric is a signal, not a gate.