Identifier Confidence Scoring Prompt Template

This playbook addresses a critical gating problem: upstream models extract identifiers like email addresses, phone numbers, and company names from unstructured text, but the extraction quality varies wildly. Some extractions are perfect, some are plausible but wrong, and some are outright fabrications. The Identifier Confidence Scoring prompt takes these extracted identifiers and produces per-field confidence scores with explicit reasoning, enabling automated decisions about which records to trust, which to flag for human review, and which to reject outright. The ideal user is a data quality engineer or pipeline operator who already has extracted identifiers and needs a calibrated trust signal—not someone who needs to extract identifiers from raw text or normalize them into a standard format.

Use this prompt when you have a batch of extracted identifier records flowing from an upstream model pipeline and you need to decide whether each record is safe to ingest into a CRM, CDP, identity resolution system, or customer-facing database. The prompt expects structured input containing the extracted identifier fields and any available source context, and it returns confidence scores (typically 0.0 to 1.0) for each field along with a brief reasoning string explaining the score. This is not a normalization prompt—it won't fix malformed emails or standardize phone numbers to E.164. Pair it with the Email Address Normalization or Phone Number Canonicalization playbooks if you need repair before scoring. The prompt works best when you provide the original source text alongside the extracted identifiers, giving the model evidence to ground its confidence assessment rather than guessing.

Do not use this prompt as a substitute for deterministic validation. If you can validate an email format with a regex or check a phone number against a known numbering plan, do that first and feed the results into the confidence prompt as additional context. Do not use this prompt for real-time, latency-sensitive flows where a 2-3 second LLM call is unacceptable—consider a lightweight classifier or heuristic scoring instead. And do not treat the confidence scores as calibrated probabilities without running your own evaluation against ground truth labels. The scores are useful for relative ranking and threshold-based gating, but they reflect the model's internal judgment, not a statistically calibrated measurement. Before deploying, run the Identifier Confidence Scoring Eval Rubric from the sibling playbooks to measure false-positive and false-negative rates at your chosen thresholds.

This prompt template is designed to be pasted directly into your application's model call. It instructs the model to act as a data quality auditor, evaluating each extracted identifier against the provided source context and returning a structured confidence score. The primary job is to prevent bad data from silently entering your systems of record by assigning a trust level to every field before ingestion.

textCopy
You are an identifier confidence auditor. Your task is to evaluate the reliability of each extracted identifier field based on the provided source text and extraction context.

## INPUT
Source Text:
[SOURCE_TEXT]

Extracted Identifiers (JSON):
[EXTRACTED_IDENTIFIERS]

## OUTPUT SCHEMA
Return a valid JSON object with the following structure. Do not include any text outside the JSON object.
{
  "fields": [
    {
      "field_name": "string",
      "extracted_value": "string | null",
      "confidence_score": 0.0-1.0,
      "confidence_level": "HIGH | MEDIUM | LOW",
      "reasoning": "string explaining the score based on source evidence",
      "requires_review": true | false
    }
  ],
  "overall_record_confidence": 0.0-1.0,
  "review_recommended": true | false,
  "summary_notes": "string"
}

## CONFIDENCE CALIBRATION
- **HIGH (0.85-1.0):** Identifier is explicitly stated in the source text with no ambiguity.
- **MEDIUM (0.5-0.84):** Identifier is present but requires inference, normalization, or has minor inconsistencies.
- **LOW (0.0-0.49):** Identifier is missing, heavily inferred, contradicts the source, or appears fabricated.

## CONSTRAINTS
- Do not invent identifiers not present in the source text.
- If a field is missing from the extracted identifiers, include it with a null value and a confidence_score of 0.0.
- Flag any identifier that appears to be a hallucination (present in extraction but not in source) with a confidence_score of 0.0 and requires_review: true.
- For email addresses, check for domain validity and common typos.
- For phone numbers, verify against expected patterns for the stated or inferred country.
- For URLs, check that the domain exists in the source context.
- If the source text is insufficient to verify any identifier, set overall_record_confidence to 0.0 and review_recommended to true.

## RISK LEVEL
[RISK_LEVEL]
- If RISK_LEVEL is "high", lower your threshold for requires_review and be more conservative in confidence scoring.
- If RISK_LEVEL is "low", you may accept reasonable normalization as MEDIUM confidence.

To adapt this template, replace the square-bracket placeholders with real values at runtime. [SOURCE_TEXT] should contain the original unstructured text the model extracted from. [EXTRACTED_IDENTIFIERS] should be the JSON output from your extraction step, even if it's malformed—this prompt can score partial or broken extractions. [RISK_LEVEL] should be set to "high", "medium", or "low" based on the downstream impact of bad data. For high-risk workflows like CRM ingestion or identity resolution, always set this to "high" and route any record with review_recommended: true to a human review queue. Before deploying, run this prompt against a golden dataset of 50-100 records with known ground truth to calibrate your confidence thresholds and measure false-positive and false-negative rates.

The Identifier Confidence Scoring prompt is not a standalone widget; it is a gating decision inside a data pipeline. The prompt receives an extracted identifier and its surrounding context, then returns a structured confidence score, reasoning, and a flag for human review. The implementation harness must enforce the output schema, handle model failures, log every scoring decision for auditability, and route low-confidence records to a review queue instead of silently ingesting them into the canonical data store.

Start by wrapping the prompt call in a function that accepts the required inputs—[IDENTIFIER], [IDENTIFIER_TYPE], [EXTRACTION_CONTEXT], and [SOURCE_DOCUMENT_SNIPPET]—and returns a typed result object. Validate the model's JSON response against a strict schema before accepting it: the confidence_score field must be a float between 0.0 and 1.0, the review_required field must be a boolean, and the reasoning field must be a non-empty string. If validation fails, retry once with an explicit error message injected into the prompt context. If the retry also fails, log the raw response and escalate the record to manual review. Use a model that supports structured output modes (JSON mode or function calling) to reduce parsing failures, but always keep the schema validator as a safety net—models can still produce valid JSON that violates field constraints.

For high-throughput pipelines, implement a scoring threshold that determines downstream routing. Records with confidence_score above 0.85 and review_required set to false can proceed to automated ingestion. Records scoring between 0.5 and 0.85 should be flagged for spot-check sampling. Anything below 0.5, or any record where review_required is true regardless of score, must enter a human review queue. Log every decision with the identifier hash (never the raw identifier if it contains PII), the confidence score, the threshold applied, the model version, and the timestamp. This log becomes your audit trail for tuning thresholds and detecting model drift. For regulated environments, ensure the review queue preserves the full prompt context so reviewers can see exactly what the model saw when making its decision.

Calibrate your thresholds against a labeled golden dataset before production deployment. Run the prompt against 200–500 records with known ground-truth correctness and measure the false-positive rate (high-confidence scores on incorrect extractions) and false-negative rate (low-confidence scores on correct extractions). Adjust the threshold until the false-positive rate is acceptable for your business risk tolerance—CRM pipelines might tolerate 5%, while healthcare or finance pipelines should target under 1%. Re-run this calibration whenever you change the prompt, the model, or the extraction upstream. The harness should make recalibration a one-command operation, not a manual spreadsheet exercise.