Case Law De-duplication

MinHash is a locality-sensitive hashing technique that estimates the Jaccard similarity between two documents by comparing their shingle sets. In legal corpora, it efficiently identifies near-duplicate opinions that differ only in header metadata, pagination, or minor OCR errors. The algorithm generates a compact signature for each document, allowing millions of comparisons in sub-linear time. A similarity threshold—typically 0.8 or higher—triggers a review or removal action.

• Shingling: Breaking text into overlapping n-gram sequences (e.g., 5-word shingles)
• Signature Matrix: A compressed representation enabling rapid pairwise comparison
• Band Partitioning: Dividing signatures into bands to tune precision-recall trade-offs

SimHash produces a fixed-length fingerprint where similar documents generate hashes with a small Hamming distance. Unlike MinHash, which estimates set overlap, SimHash is designed for cosine similarity detection in high-dimensional spaces. For legal de-duplication, it excels at catching documents that share substantial textual content but have been reformatted, re-paragraphed, or had boilerplate text inserted. Its compact fingerprints enable efficient storage and lookup in a distributed index.

• Hamming Distance: The number of bit positions where two hashes differ
• Fingerprint Length: Typically 64 or 128 bits for legal-scale corpora
• Bucketization: Grouping fingerprints by bit prefixes to prune the search space

Before any fuzzy matching occurs, legal documents must be normalized to a canonical citation form. This process strips away vendor-specific formatting, parallel citations, and regional reporter variations to extract the core authority reference. For example, '347 U.S. 483' and '347 U.S. 483, 74 S.Ct. 686' resolve to the same canonical identifier. This step prevents a single case from appearing multiple times in the corpus under different citation strings.

• Reporter Mapping: Standardizing across West, Lexis, and official reporters
• Vendor Stripping: Removing database-specific metadata and pagination
• Parallel Citation Collapse: Reducing multiple citation forms to a single key

For the hardest class of duplicates—paraphrased opinions or cases that have been substantially rewritten by different courts—semantic embeddings provide a final defense. A legal embedding model maps each document to a dense vector, and cosine similarity or clustering algorithms like DBSCAN group semantically identical content. This catches duplicates that share no significant n-gram overlap but convey the same judicial reasoning.

• Dense Vector Representation: 768- to 4096-dimensional embeddings
• DBSCAN: Density-based clustering that does not require pre-specifying the number of clusters
• Cosine Threshold: Typically 0.95+ for near-duplicate flagging

De-duplication must operate across data splits, not just within them. A case appearing in the training set must be excluded from validation and test sets to prevent benchmark leakage. This requires maintaining a global index of document fingerprints and checking every new document against it before assigning it to a split. Failure to do so inflates evaluation metrics and masks a model's true generalization capability.

• Global Fingerprint Index: A persistent store of all seen document hashes
• Split-Aware Filtering: Checking against train, validation, and test indices simultaneously
• Leakage Audit: Retrospective analysis to verify no contamination occurred

In legal AI, a subtle form of contamination arises when a model trained on documents up to 2023 is evaluated on a case from 2019 that was later cited or summarized in a 2022 document in the training set. Temporal leakage occurs when future knowledge of a case seeps backward through citations. Effective de-duplication pipelines must track the publication date of each document and ensure that no evaluation document is referenced, summarized, or quoted in any training document with a later date.

• Citation Graph Traversal: Following citation links to detect indirect exposure
• Date-Anchored Splits: Partitioning data strictly by chronological cutoffs
• Quotation Fingerprinting: Detecting embedded text from later cases in earlier training documents

A critical failure in legal AI where evaluation data is inadvertently included in the pre-training corpus. When a model encounters test questions during training, its performance metrics become invalid—the model is no longer reasoning but memorizing answers. De-duplication against benchmark datasets like LexGLUE is the primary defense. Even partial overlap, such as a case citation appearing in both training and test sets, can artificially inflate scores by 10-30 percentage points.

An algorithm that efficiently estimates the Jaccard similarity between document pairs without comparing every document exhaustively. MinHash creates compact 'sketches' of each legal opinion by hashing its n-grams, then uses locality-sensitive hashing (LSH) to bucket similar sketches together. For a corpus of millions of cases, this reduces pairwise comparisons from O(n²) to near-linear time, making large-scale de-duplication computationally feasible.

Legal documents rarely duplicate exactly. Common near-duplicate patterns include:

• Header variations: Same opinion with different court headers or docket numbers
• Quotation chains: Opinions that extensively quote each other, creating high textual overlap
• Publisher variants: The same case from Westlaw, LexisNexis, and CourtListener with different formatting Setting the similarity threshold (typically 0.8-0.95 Jaccard) is critical—too aggressive removes legitimate precedent citations; too lenient leaves contamination.

A dimensionality reduction technique that generates a fixed-length binary fingerprint for each document. Documents with small Hamming distances between their fingerprints are near-duplicates. SimHash is particularly effective for legal text because it is resilient to minor edits—a judicial opinion with corrected typos or added parallel citations will still produce a nearly identical fingerprint, enabling robust de-duplication across publisher variants.

A lossless, exact-match approach that identifies all duplicated substrings above a minimum length across the entire corpus. Unlike fuzzy hashing methods, suffix arrays catch verbatim paragraph reuse—common when the same legal standard is recited across hundreds of opinions. This method is computationally expensive but guarantees removal of exact textual duplicates that MinHash might miss due to sketch collisions.

A domain-specific technique that leverages the legal citation network to identify duplicates. Two documents citing the same authority with identical quotation patterns are likely the same case. By constructing a bipartite graph of documents and their cited cases, structural graph matching algorithms can detect duplicates that text-based methods miss—such as opinions that paraphrase rather than copy text but represent the same judicial decision.