Extraction Reproducibility Audit Prompt

This prompt is for audit teams, data platform engineers, and ML reliability engineers who need to prove that an extraction pipeline produces consistent results across repeated runs. The core job-to-be-done is detecting non-deterministic behavior: fields that change value when the same document is processed multiple times with identical parameters, model version, and temperature settings. You should use this prompt when you are qualifying a new extraction prompt for production, validating a model upgrade, or responding to an incident where downstream consumers report inconsistent extracted values. The ideal user has access to a document corpus, a stable extraction prompt, and the ability to run multiple inference passes with controlled settings.

Do not use this prompt when you are debugging a single failed extraction, measuring accuracy against ground truth, or checking schema compliance. Those jobs belong to the Extraction Error Categorization Prompt, Golden Dataset Validation Prompt, and Schema Drift Detection Prompt respectively. This prompt is specifically for reproducibility: it assumes the extraction logic is already defined and the question is whether the model applies that logic consistently. You need at least three repeated extraction runs per document to produce a meaningful variance analysis. The prompt works best with structured JSON extraction outputs where field-level comparison is straightforward; free-text extraction or narrative summaries will produce noisy and less actionable reproducibility scores.

Before running this audit, lock down your extraction prompt version, model version, and sampling parameters. Any drift in these inputs will invalidate the reproducibility measurement. The output is a per-field reproducibility score with variance details, not a binary pass/fail. Use the results to identify fields that need tighter prompt constraints, lower temperature, or a switch to a more deterministic model. For high-stakes pipelines where non-determinism could cause compliance or financial impact, pair this audit with the Confidence Score Calibration Audit Prompt and require human review of fields that show variance above your defined threshold.

This prompt template is designed to be executed multiple times against the same source document and extraction parameters. It instructs the model to compare extraction outputs across runs, identify fields that produce different values despite identical inputs, and produce a structured reproducibility report. The template uses square-bracket placeholders for all variable inputs, making it ready to drop into an automated audit pipeline.

textCopy
You are an extraction reproducibility auditor. Your task is to compare multiple extraction runs performed on the same source document with identical parameters and flag any non-deterministic behavior.

## INPUTS
- Source Document: [SOURCE_DOCUMENT]
- Extraction Schema: [EXTRACTION_SCHEMA]
- Extraction Run Outputs: [EXTRACTION_RUNS]
  Provide the full output from each run, labeled with a run identifier (e.g., Run-1, Run-2, Run-3).
- Minimum Runs Required: [MIN_RUNS]
- Reproducibility Threshold: [REPRODUCIBILITY_THRESHOLD]
  Fields with agreement below this percentage are flagged as non-reproducible.

## INSTRUCTIONS
1. Parse all extraction run outputs against the provided schema.
2. For each field in the schema, compare the extracted value across all runs.
3. Classify each field into one of these reproducibility categories:
   - FULLY_REPRODUCIBLE: All runs produced identical values.
   - PARTIALLY_REPRODUCIBLE: Some runs differ, but agreement meets or exceeds the threshold.
   - NON_REPRODUCIBLE: Agreement falls below the threshold.
   - INSUFFICIENT_DATA: Fewer runs available than the minimum required.
4. For any field that is not FULLY_REPRODUCIBLE, document the specific variance:
   - Which runs disagreed
   - The differing values produced
   - Whether the variance is structural (different types), semantic (different meaning), or surface-level (formatting, whitespace, casing)
5. Compute a per-field reproducibility score as: (number of runs agreeing on the most common value) / (total runs) * 100.
6. Compute an overall document reproducibility score as the mean of all per-field scores.

## OUTPUT SCHEMA
Return a JSON object with this structure:
{
  "audit_metadata": {
    "audit_timestamp": "ISO-8601 timestamp of audit execution",
    "total_runs_compared": <integer>,
    "schema_field_count": <integer>,
    "overall_reproducibility_score": <float 0-100>
  },
  "field_results": [
    {
      "field_path": "dot.notation.path.to.field",
      "reproducibility_category": "FULLY_REPRODUCIBLE | PARTIALLY_REPRODUCIBLE | NON_REPRODUCIBLE | INSUFFICIENT_DATA",
      "reproducibility_score": <float 0-100>,
      "run_values": {
        "Run-1": <extracted value or null>,
        "Run-2": <extracted value or null>
      },
      "variance_type": "NONE | STRUCTURAL | SEMANTIC | SURFACE | null",
      "variance_detail": "Description of what differed, or null if fully reproducible",
      "dominant_value": "The most common value across runs, or null",
      "dominant_value_run_count": <integer>
    }
  ],
  "non_reproducible_summary": {
    "total_non_reproducible_fields": <integer>,
    "total_partially_reproducible_fields": <integer>,
    "flagged_field_paths": ["list", "of", "field", "paths"],
    "recommended_actions": ["List of remediation suggestions such as: add explicit formatting constraints, check temperature settings, review ambiguous schema descriptions, consider post-extraction normalization"]
  }
}

## CONSTRAINTS
- Do not hallucinate run values. Only report values actually present in the provided extraction outputs.
- If a field is missing from some runs but present in others, treat the missing runs as having a null value for that field.
- Surface-level variance (trailing whitespace, Unicode normalization differences, case differences in clearly case-insensitive fields) should be noted but may be classified as FULLY_REPRODUCIBLE if the semantic content is identical. Document your reasoning.
- If [RISK_LEVEL] is HIGH, flag any NON_REPRODUCIBLE field as requiring human review before downstream consumption.

To adapt this template, replace the square-bracket placeholders with your specific extraction context. The [EXTRACTION_RUNS] placeholder should receive the complete JSON output from each extraction run, keyed by run identifier. For production use, set [MIN_RUNS] to at least 3 to avoid false confidence from a single agreeing pair. The [REPRODUCIBILITY_THRESHOLD] should be tuned to your risk tolerance—90% is a common starting point for high-stakes extraction, while 70% may be acceptable for exploratory analysis. The [RISK_LEVEL] flag controls whether the prompt adds human-review escalation language to the output. Before deploying, validate that your extraction schema is expressed in a format the model can reliably parse against the run outputs—ambiguous field descriptions in the schema are a common source of false reproducibility flags.

The Extraction Reproducibility Audit Prompt is designed to be run multiple times against the same input document and extraction parameters to detect non-deterministic behavior. In a production harness, you should execute the extraction prompt at least three times (odd number recommended for majority-vote tiebreaking) with identical inputs, model version, temperature, and seed where supported. Each run must be isolated—do not pass prior run outputs as context—to prevent the model from converging on a previous answer. Store each run's raw output alongside the input hash, timestamp, model version, and any sampling parameters for later comparison.

Wire the prompt into a job queue or workflow orchestrator (e.g., Temporal, Airflow, or a simple Lambda with Step Functions) that accepts a document payload and extraction schema contract. The harness should: (1) validate that the input document and schema are unchanged between runs by comparing content hashes; (2) execute the extraction prompt N times with the same request shape; (3) collect all outputs and pass them to the reproducibility analysis step. For the analysis, use a structured comparison script—not another LLM call—to compute field-level agreement rates, flag fields where values diverge across runs, and calculate a reproducibility score per field. A field is reproducible if all runs produce identical values; partial agreement should be quantified as a percentage. For high-risk pipelines (finance, healthcare, legal), require a human review step when any critical field shows less than 100% agreement across runs.

Model choice matters significantly for reproducibility. Prefer models with low-temperature deterministic modes and documented seed support (e.g., GPT-4o with seed parameter, Claude with temperature=0). Even with these settings, some models exhibit residual non-determinism due to floating-point variance or batch scheduling. Log the full model identifier, temperature, seed, and any API-level parameters in the audit trail. If your extraction pipeline uses a model router or fallback chain, lock the reproducibility audit to a single model version to avoid conflating model variance with prompt variance. For RAG-augmented extraction, ensure the retrieval step is also deterministic—use the same embedding model, index snapshot, and retrieval parameters across runs, and include the retrieved context hash in the audit log.

Validation and evals should be built into the harness, not applied after the fact. Before the reproducibility analysis runs, validate each extraction output against the expected schema using a JSON Schema validator. Reject malformed outputs and retry with the same parameters (up to a configured retry limit) before marking a run as failed. For eval, maintain a golden set of documents with known extraction outputs and run the reproducibility audit against them periodically. A drop in reproducibility scores on the golden set signals model behavior drift, prompt degradation, or infrastructure changes. Wire these metrics into your observability stack (e.g., Datadog, Grafana) with alerts when reproducibility falls below a defined threshold. Avoid running reproducibility audits on every production document—sample periodically or trigger on schema changes, model upgrades, or prompt version bumps to control cost while maintaining coverage.