Duplicate Detection

The foundational layer of duplicate detection that generates a unique digital fingerprint for a document at the binary level. Cryptographic hashing algorithms like SHA-256 process the entire file to produce a fixed-size string. If two documents produce the same hash, they are exact, bit-for-bit identical copies.

• Mechanism: Computes a checksum on the raw file bytes
• Strength: Extremely fast and computationally cheap
• Limitation: Fails to detect near-duplicates; a single character difference produces a completely different hash
• Use Case: Identifying exact re-transmissions of the same PDF or image file

A locality-sensitive hashing technique that identifies documents with high textual similarity even when they are not identical. SimHash computes a fingerprint where similar documents produce hashes with a small Hamming distance between them.

• Mechanism: Tokenizes text, weights features, and generates a compact binary fingerprint
• Strength: Catches clinically identical reports that differ only in timestamps, headers, or minor whitespace
• Key Metric: Hamming distance threshold (e.g., ≤ 3 bits) defines similarity tolerance
• Use Case: Flagging a radiology report that was ingested twice with different metadata wrappers

Moves beyond surface-level text comparison to identify documents conveying the same clinical fact even when phrased differently. This approach uses dense vector embeddings generated by a fine-tuned language model to capture semantic meaning.

• Mechanism: Encodes the clinical text into a high-dimensional vector; computes cosine similarity between embeddings
• Strength: Detects paraphrased or summarized versions of the same clinical event
• Key Component: A healthcare-specific embedding model fine-tuned on clinical narratives
• Use Case: Identifying that a discharge summary and a transfer note describe the same patient encounter

A deterministic rules engine that compares structured header fields before engaging computationally expensive content analysis. This tiered approach filters out obvious non-duplicates early.

• Fields Compared: Patient MRN, document type, author, facility, and service date
• Logic: If two documents share the same MRN, document type, and service date, they are candidates for deeper content comparison
• Strength: Reduces the search space dramatically, improving pipeline throughput
• Use Case: Pre-filtering a batch of 10,000 incoming reports to only compare those for the same patient on the same day

Constrains the duplicate search scope to a clinically relevant time window rather than scanning the entire patient history. A duplicate document is only clinically meaningful if it arrives within a defined proximity to the original.

• Typical Window: 30 to 90 days surrounding the document's service date
• Rationale: An identical report arriving years later is likely a legitimate re-issue, not a duplicate
• Implementation: Indexes documents by patient ID and service date for range queries
• Use Case: Ensuring a re-sent report from a year ago is not incorrectly flagged as a duplicate

Defines the automated and manual workflows triggered when a duplicate is detected. The system must not simply delete data but instead link, overlay, or quarantine the redundant entry while maintaining an immutable audit trail.

• Actions: Auto-reject, overlay metadata, link as a superseded version, or route to an exception queue for manual review
• Compliance: Every resolution action is logged with a timestamp, user ID, and rationale
• Principle: The original document is never deleted; the duplicate is marked with a lifecycle state of 'superseded' or 'duplicate'
• Use Case: A human reviewer confirms two documents are duplicates and links them, preserving both for legal health record requirements

A computational method that generates a unique digital fingerprint for a document to efficiently identify exact duplicates at the binary level. This technique uses cryptographic hash functions like SHA-256 to produce a fixed-size digest of the document's entire content.

• Detects bit-for-bit identical files regardless of filename or metadata
• Operates in O(1) time for lookup after initial hash computation
• Cannot identify near-duplicates or semantically identical documents with minor formatting differences
• Commonly used as a first-pass filter before more computationally expensive fuzzy matching

A technique that generates a unique content-based identifier for a document to detect duplicates or track versions independent of file name or metadata. Unlike simple hashing, fingerprinting can be designed to be robust to minor variations.

• SimHash and MinHash algorithms enable locality-sensitive hashing for near-duplicate detection
• Can identify documents that share a high percentage of n-gram overlap
• Essential for detecting the same clinical report ingested through different pathways (e.g., HL7 feed vs. direct upload)
• Forms the basis for document versioning and amendment tracking

A text segmentation strategy that splits documents based on semantic boundaries, such as section headers, rather than arbitrary character counts. This is critical for duplicate detection because it enables section-level comparison.

• Identifies structural elements like 'Impression', 'Findings', and 'Clinical History'
• Prevents false positives when only the header metadata differs between two copies of the same report
• Enables targeted deduplication logic: flag if Impression + Findings match, even if administrative sections differ
• Uses clinical section classifiers trained on document type ontologies

A patient matching approach that uses statistical likelihood scores to link records, accounting for variations, typos, and missing data in demographics. Applied to document deduplication, it enables fuzzy content matching beyond exact hashing.

• Uses algorithms like Fellegi-Sunter to compute match probabilities
• Accounts for OCR errors, transcription variations, and minor formatting differences
• Assigns confidence scores that feed into automated decision thresholds
• Critical for matching documents that are semantically identical but not bit-for-bit identical

A filtering mechanism that routes AI predictions with low probability scores to a manual review queue, ensuring high accuracy for automated decisions. In duplicate detection, this determines the boundary between auto-resolution and human review.

• High-confidence matches (>98%) are automatically suppressed from the patient record
• Medium-confidence matches (85-98%) are flagged for clinician verification
• Low-confidence matches (<85%) are treated as new documents
• Thresholds are configurable by document type and clinical risk profile

The immutable recording of all system interactions, data modifications, and access events related to a clinical document for compliance and security. Every duplicate detection decision must be fully auditable.

• Logs which document was flagged as a duplicate and why
• Records the matching algorithm, confidence score, and timestamp
• Tracks human overrides when a reviewer accepts or rejects a duplicate flag
• Essential for HIPAA compliance and medicolegal defensibility of the patient record