Inferensys

Prompt

Structured Diff Output Prompt with Source Attribution

A practical prompt playbook for generating structured, machine-readable diffs with source attribution in production document comparison pipelines.
Wide-angle shot of a modern WeWork open floor plan with creative walls covered in AI system architecture diagrams, product team collaborating in standing desk area with industrial lighting.
PROMPT PLAYBOOK

When to Use This Prompt

Understand the job-to-be-done, ideal user, required context, and when not to use the Structured Diff Output Prompt with Source Attribution.

This prompt is designed for document pipeline engineers who need to compare two document versions and produce a structured, machine-readable diff. It is not a redline renderer or a human-facing summary tool. Use it when your application must ingest comparison output into a database, trigger downstream workflows, or feed a review queue. The prompt generates a JSON array of change objects, each with a change type, before/after text, a section path for source attribution, and a confidence score. It is built for legal, compliance, and contract management systems where traceability and schema compliance are non-negotiable.

The ideal user is an engineering team building a document version analysis pipeline. Required context includes two complete document versions with identifiable section structures—the prompt relies on section paths to attribute changes. You must provide a strict output schema, such as a JSON Schema definition, to enforce field types, required fields, and enum values for change types. The prompt expects documents where differences are textual and structural, not purely visual or formatting-based. If your documents lack section headings, numbered clauses, or other structural markers, the source attribution will degrade, and you should consider a layout-aware extraction step before invoking this prompt.

Do not use this prompt for free-text summarization, rendering visual diffs, or comparing documents where section structure is absent. It is also unsuitable for comparing more than two documents simultaneously—for multi-document reconciliation, use a dedicated cross-file inconsistency prompt. Avoid this prompt when the primary goal is generating a human-readable redline; the output is a JSON array meant for machine consumption, not a marked-up document. If your use case involves scanned PDFs with OCR noise, pre-process with an OCR-aware extraction prompt and validate text quality before comparison, as degraded text will produce false-positive changes and unreliable confidence scores. Always pair this prompt with a validation step that checks schema compliance, citation accuracy, and confidence thresholds before the output enters any downstream system.

PRACTICAL GUARDRAILS

Use Case Fit

Where the Structured Diff Output Prompt with Source Attribution works well, where it breaks, and the operational prerequisites for production use.

01

Good Fit: Machine-Readable Version Reconciliation

Use when: You need a structured JSON diff between two document versions for downstream ingestion into a contract management system, compliance database, or automated review pipeline. Guardrail: Define the output schema (change type, before/after text, section path, confidence) before writing the prompt, and validate every field against that schema in your application layer.

02

Bad Fit: Unstructured Narrative Summaries

Avoid when: The primary requirement is a prose summary of what changed for a human reader. This prompt is designed for structured, machine-actionable output. Guardrail: If a narrative is also needed, generate it in a separate step from the structured diff to avoid format contamination and hallucinated prose.

03

Required Inputs: Clean, Comparable Text Pairs

Risk: Feeding raw OCR output, heavily scanned PDFs, or documents with significant layout noise will produce garbage diffs with hallucinated changes. Guardrail: Pre-process documents through layout-aware extraction and OCR cleanup pipelines. Validate text quality before comparison. If confidence in the source text is low, flag the diff as unreliable rather than returning low-confidence changes as facts.

04

Operational Risk: Citation Drift Under Formatting Changes

Risk: Formatting-only changes (reflow, pagination, font size) can shift section and page citations, causing the model to report spurious location changes or miss real content changes. Guardrail: Normalize document structure before comparison. Use semantic section identifiers (clause numbers, heading text) rather than page numbers alone. Add a validation step that checks whether a reported change is a content change or a formatting artifact.

05

Operational Risk: Hallucinated Clause Text

Risk: The model may generate plausible-sounding but fabricated clause text in the after field, especially when the source documents are ambiguous or the change is implied rather than explicit. Guardrail: Implement a post-generation grounding check that verifies every after text string exists verbatim in the target document. Flag any diff entry that fails this check for human review.

06

Scale Risk: High Token Cost on Large Documents

Risk: Comparing two 200-page contracts inline consumes significant context window and token budget, leading to cost overruns and potential mid-document truncation. Guardrail: Chunk documents by section before comparison. Run the diff prompt per section pair and assemble results. This also improves citation accuracy and makes failures easier to isolate and retry.

PROMPT PLAYBOOK

Copy-Ready Prompt Template

A copy-ready prompt template for generating a structured JSON diff with source attribution between two document versions.

This prompt template is designed to be dropped into a system that enforces JSON-only output and schema compliance. It instructs the model to compare two document versions and produce a machine-readable diff. The output includes the type of change, the exact before and after text, a precise section path for source attribution, and a confidence score. Before using this template, ensure you have extracted clean, plain-text representations of both document versions, as layout artifacts will degrade comparison accuracy.

text
You are a document comparison engine. Your task is to compare the [PREVIOUS_VERSION] and [CURRENT_VERSION] of a document and output a structured JSON diff. You must attribute every change to its exact location using a section path.

Follow these rules strictly:
1. Identify all additions, deletions, and modifications.
2. For each change, extract the exact text from both versions. If text was added, the 'before' field must be an empty string. If text was deleted, the 'after' field must be an empty string.
3. Construct a 'section_path' for each change using the format "Document > Section > Subsection". Do not use page numbers.
4. Assign a 'confidence' score between 0.0 and 1.0. Use a lower score for ambiguous changes or formatting-only differences.
5. Classify the 'change_type' as one of: "addition", "deletion", "modification", or "formatting".
6. If there are no material differences, return an empty array.
7. Do not include any commentary outside the JSON object.

[PREVIOUS_VERSION]:
[INSERT_PREVIOUS_DOCUMENT_TEXT]

[CURRENT_VERSION]:
[INSERT_CURRENT_DOCUMENT_TEXT]

[OUTPUT_SCHEMA]:
{
  "diffs": [
    {
      "change_type": "modification",
      "before": "The quick brown fox",
      "after": "The quick red fox",
      "section_path": "Article I > Definitions > Section 1.2",
      "confidence": 0.98
    }
  ]
}

To adapt this prompt, replace the [INSERT_PREVIOUS_DOCUMENT_TEXT] and [INSERT_CURRENT_DOCUMENT_TEXT] placeholders with your pre-processed text. For high-stakes legal or compliance workflows, you should add a [CONSTRAINTS] section that explicitly defines what constitutes a 'material' change versus a formatting change. After receiving the output, validate the JSON against the provided schema and implement a secondary check that verifies each section_path actually exists in the source documents to catch hallucinated citations.

IMPLEMENTATION TABLE

Prompt Variables

Each placeholder must be populated before the prompt is sent to the model. Validation notes describe what the model expects and common failure points.

PlaceholderPurposeExampleValidation Notes

[DOCUMENT_A]

Base version of the document for comparison

contract_v2_2025-03-15.pdf

Must be parseable text. Scanned docs require OCR preprocessing. Null not allowed. Failure mode: model hallucinates changes when input is empty or unreadable.

[DOCUMENT_B]

Newer version to compare against the base

contract_v3_2025-06-01.pdf

Must be parseable text. If identical to DOCUMENT_A, output should contain zero changes. Failure mode: model invents formatting-only diffs when documents are semantically identical.

[DOCUMENT_TYPE]

Document category for context-aware comparison

Master Services Agreement

Must match a known type in your taxonomy. Affects clause boundary detection. Failure mode: generic 'document' type causes missed section headers and merged clauses in diff output.

[COMPARISON_SCOPE]

Scope of comparison: full document or specific sections

Sections 3-12 only

Use section numbers or 'full'. Partial scope must match actual document structure. Failure mode: mismatched section references cause model to compare wrong regions or miss changes outside scope.

[CHANGE_THRESHOLD]

Minimum significance level for reporting a change

0.7

Float between 0.0 and 1.0. Lower values report formatting changes; higher values suppress minor wording differences. Failure mode: threshold too low floods output with whitespace diffs; too high suppresses material clause changes.

[OUTPUT_SCHEMA_VERSION]

Schema version for the structured diff output

2.1

Must match your downstream parser version. Schema changes between versions can break field expectations. Failure mode: version mismatch causes parser rejection even when diff content is correct.

[CITATION_GRANULARITY]

Level of source attribution required

section.paragraph

Options: page, section, paragraph, line, or sentence. Must be achievable given document structure. Failure mode: requesting line-level citations on a document without line numbers causes fabricated citations.

[CONFIDENCE_FORMAT]

How confidence scores are expressed in output

decimal

Options: decimal (0.0-1.0), percentage (0-100), or categorical (low/medium/high). Must match your eval pipeline expectations. Failure mode: format mismatch causes all confidence fields to fail validation even when values are correct.

PROMPT PLAYBOOK

Implementation Harness Notes

How to wire the structured diff prompt into a production document comparison pipeline with validation, retries, and observability.

The structured diff prompt is not a standalone tool—it is one step in a document comparison pipeline. In production, you will call this prompt after extracting and normalizing text from both document versions, and before storing or displaying the diff results. The prompt expects pre-extracted text segments with section path metadata, not raw PDFs or scanned images. This means your pipeline must handle layout-aware extraction, OCR cleanup, and section hierarchy parsing upstream. The prompt's output—a JSON array of change objects with change_type, before_text, after_text, section_path, and confidence—must then pass through validation, deduplication, and citation accuracy checks before it reaches any downstream system or user interface.

Wire the prompt into your application with a validation-first architecture. After receiving the model's JSON response, run a schema validator that checks required fields, enum values for change_type (only added, removed, modified, moved, formatting_only), and confidence score ranges (0.0 to 1.0). Next, run a citation accuracy check: for each change object, verify that the before_text and after_text substrings actually appear at the claimed section_path in the source documents. This prevents hallucinated diffs where the model invents changes that do not exist in either document. If citation accuracy falls below a threshold (start with 95%), flag the entire diff for human review rather than silently accepting it. For high-risk domains like legal or compliance, add a human approval queue that surfaces low-confidence changes (confidence < 0.7) and changes where the before_text or after_text cannot be verified in the source documents.

Build retry logic around the model call itself. If the response fails JSON parsing, retry once with the same prompt plus the parse error message appended as a correction instruction. If the second attempt also fails, log the raw response and escalate. For formatting_only changes, apply a deduplication pass: if the model reports a formatting change at the same section path as a content change, keep only the content change and discard the formatting-only entry to avoid noise. Choose a model with strong structured output support—GPT-4o or Claude 3.5 Sonnet with JSON mode enabled—and set temperature=0 for deterministic diffs. Log every prompt call with the input document hashes, model version, raw response, validation results, and final accepted diff count. This trace data becomes essential when debugging false positives, missed changes, or format drift after a model upgrade.

IMPLEMENTATION TABLE

Expected Output Contract

The JSON schema, field definitions, and validation rules that the model response must satisfy before it enters your application. Each row defines a required or optional element of the structured diff output.

Field or ElementType or FormatRequiredValidation Rule

diff_entries

Array of objects

Must be a non-null array. If no changes are detected, return an empty array. Schema check: type assertion before iteration.

diff_entries[].change_type

Enum: [added, removed, modified, unchanged]

Must be one of the four allowed values. Reject any response containing an unrecognized change_type. If the model returns formatting-only, map to unchanged with a note.

diff_entries[].before_text

String or null

Required for removed and modified entries. Must be null for added entries. Null check: if change_type is removed or modified and before_text is null or empty, flag for repair.

diff_entries[].after_text

String or null

Required for added and modified entries. Must be null for removed entries. Null check: if change_type is added or modified and after_text is null or empty, flag for repair.

diff_entries[].section_path

String

Must contain a document identifier and a navigable path (e.g., [DOC_ID] > Section 4.2(a)). Citation check: verify the path exists in the source document structure. Reject fabricated section numbers.

diff_entries[].confidence

Number (0.0-1.0)

Must be a float between 0.0 and 1.0 inclusive. Confidence threshold: entries below [MIN_CONFIDENCE] should be routed for human review rather than automatically ingested.

diff_entries[].formatting_only

Boolean

Must be true if the change involves only whitespace, font, or layout differences with no semantic content change. If true, change_type should be unchanged. Audit check: sample 10% of formatting_only=true entries for false negatives.

processing_metadata.model_version

String

If present, must match the expected [MODEL_IDENTIFIER] pattern. Log mismatch as a warning. Useful for traceability when routing diffs across model upgrades.

PRACTICAL GUARDRAILS

Common Failure Modes

Structured diff prompts fail in predictable ways. These are the most common production failure modes for JSON diff generation with source attribution, and how to guard against them before they corrupt your document pipeline.

01

Hallucinated Changes

What to watch: The model fabricates additions, deletions, or modifications that don't exist in either document. This is especially common when documents are long, similar, or contain boilerplate language the model expects to see changed. Guardrail: Require exact citation to section, paragraph, and line number for every change. Run a post-extraction validation that verifies the cited source text exists in the original documents. Flag and discard changes where the before/after text cannot be matched to the source files.

02

Formatting-Only Noise

What to watch: The model reports whitespace changes, line breaks, font differences, or margin adjustments as substantive modifications. This floods the diff output with meaningless entries and obscures real changes. Guardrail: Add explicit instructions to ignore formatting-only changes. Include a change_type field with values like substantive, formatting, structural, and definitional. Post-process to filter out formatting entries or route them to a separate low-priority review queue.

03

Citation Drift and Wrong Section References

What to watch: The model attributes a change to Section 4.2 when it actually occurred in Section 4.3, or cites a paragraph number that doesn't exist. Citation errors destroy trust in the output and make human verification harder. Guardrail: Include the full section path in the output schema (section_path, paragraph_number, line_range). Run a citation accuracy check that confirms every cited location exists in the source document and contains text matching the reported change. Reject or flag changes with unresolvable citations.

04

Missed Changes in Long Documents

What to watch: The model misses substantive changes buried deep in long documents, especially in schedules, exhibits, appendices, or sections beyond the first few pages. Attention dilution causes silent false negatives. Guardrail: Chunk documents by section or logical boundary before comparison. Run the diff prompt on each chunk pair independently, then merge results. Add a completeness check that compares the total number of sections processed against the document's table of contents. Flag sections that weren't compared.

05

Confidence Score Inflation

What to watch: The model assigns high confidence scores to hallucinated or uncertain changes, making it impossible to triage by confidence. Every change gets 0.95 regardless of actual certainty. Guardrail: Require the model to justify its confidence score with specific evidence (e.g.,

06

Schema Drift Under Document Variation

What to watch: The output schema breaks when documents have unusual structures—missing sections, non-standard numbering, tables that span pages, or mixed content types. The model either omits required fields or invents new ones. Guardrail: Define a strict JSON schema with required fields and enum constraints. Validate every output against the schema before accepting it. Add retry logic with the schema validation error fed back to the model. If retries fail, fall back to a simpler extraction format and flag for manual review.

IMPLEMENTATION TABLE

Evaluation Rubric

Run these checks against a golden dataset of document pairs with known changes before shipping the Structured Diff Output Prompt to production. Each criterion targets a specific failure mode observed in diff generation.

CriterionPass StandardFailure SignalTest Method

Change Detection Recall

=95% of ground-truth changes appear in output with correct [CHANGE_TYPE]

Missing additions, deletions, or modifications; silent omissions in long documents

Compare output change list against manually annotated golden diff; count false negatives

Change Detection Precision

<=5% of reported changes are false positives (hallucinated or formatting-only)

Output flags unchanged text as modified; reports styling shifts as semantic changes

Review each output change against source documents; flag any change not in golden set

Citation Accuracy

100% of citations resolve to correct [SECTION_PATH] and [PAGE_NUMBER] in source documents

Section number off by one; page reference points to wrong location; path format inconsistent

Parse each citation, navigate to referenced location in source PDF, verify text match

Before/After Text Fidelity

Exact substring match between [BEFORE_TEXT]/[AFTER_TEXT] and source documents for >=98% of changes

Paraphrased or truncated text; extra whitespace; missing punctuation; ellipsis insertion

Extract quoted spans from output, perform exact string search in source documents at cited location

Confidence Score Calibration

[CONFIDENCE] >=0.9 correlates with correct changes; [CONFIDENCE] <0.6 correlates with errors or ambiguity

High-confidence output contains errors; low-confidence output is consistently correct; scores are uniform

Bin outputs by confidence range, compute precision per bin; expect monotonic relationship

Schema Compliance

Output validates against [OUTPUT_SCHEMA] with zero structural errors across all test cases

Missing required fields; wrong enum values for [CHANGE_TYPE]; malformed JSON; extra fields

Run JSON Schema validator against every output; count structural violations per test case

Formatting-Only Change Suppression

Zero formatting-only changes reported unless [INCLUDE_FORMATTING] is explicitly true

Whitespace diffs, font changes, or margin shifts appear as content changes in output

Include document pairs with only formatting changes in golden set; verify empty or flagged output

Multi-Change Ordering

Changes appear in document order matching source layout for >=95% of test cases

Changes from page 5 appear before page 2; section ordering inverted; no consistent sort

Extract [SECTION_PATH] and [PAGE_NUMBER] from each change, verify monotonic progression through document

ADAPTATION OPTIONS

Adapt This Prompt

How to adapt

Use the base prompt with a frontier model (GPT-4o, Claude 3.5 Sonnet) and a simple JSON schema. Skip confidence scoring and focus on change_type, before_text, after_text, and section_path. Accept raw text input instead of requiring PDF parsing.

code
[SYSTEM]
You are a document comparison engine. Compare [DOCUMENT_A] and [DOCUMENT_B]. Output a JSON array of changes with keys: change_type, before_text, after_text, section_path.

Watch for

  • Formatting-only changes flagged as substantive
  • Missing section paths when headings are inconsistent
  • Hallucinated changes when documents are substantially different
Prasad Kumkar

About the author

Prasad Kumkar

CEO & MD, Inference Systems

Prasad Kumkar is the CEO & MD of Inference Systems and writes about AI systems architecture, LLM infrastructure, model serving, evaluation, and production deployment. Over 5+ years, he has worked across computer vision models, L5 autonomous vehicle systems, and LLM research, with a focus on taking complex AI ideas into real-world engineering systems.

His work and writing cover AI systems, large language models, AI agents, multimodal systems, autonomous systems, inference optimization, RAG, evaluation, and production AI engineering.