Jaccard Index

The Jaccard Index is defined as the size of the intersection divided by the size of the union of two sets.

• Formula: J(A, B) = |A ∩ B| / |A ∪ B|
• Range: The result is always between 0 (no similarity) and 1 (identical sets).
• Example: For sets A = {apple, banana, cherry} and B = {banana, cherry, date}, the intersection is {banana, cherry} (size 2) and the union is {apple, banana, cherry, date} (size 4). The Jaccard Index is 2/4 = 0.5.

To apply the Jaccard Index to text, documents are first decomposed into shingles (n-grams of words or characters).

• Tokenization: A document is broken into a set of contiguous subsequences.
• Fingerprinting: Each shingle is hashed to create a compact set representation.
• Efficiency: Comparing the Jaccard similarity of shingle sets is computationally cheaper than full string matching, enabling rapid identification of near-duplicate content across large corpora.

The Jaccard Distance measures dissimilarity and is calculated as 1 minus the Jaccard Index.

• Formula: D(A, B) = 1 - J(A, B)
• Interpretation: A distance of 0 means the sets are identical; a distance of 1 means they share no elements.
• Application: This metric is often used as a distance function in clustering algorithms like k-means to group similar documents together for canonicalization.

Calculating the exact Jaccard Index for all pairs of documents is computationally prohibitive at scale. MinHash is a technique that estimates the Jaccard Index efficiently.

• Mechanism: It generates a compact signature for each set by applying multiple hash functions and recording the minimum hash value for each.
• Property: The probability that two sets have the same MinHash value equals their Jaccard Index.
• Use Case: MinHash enables the use of Locality-Sensitive Hashing (LSH) to quickly find candidate near-duplicate pairs without performing all pairwise comparisons.

The standard Jaccard Index treats all elements equally. The Weighted Jaccard Index extends the concept to sets where elements have associated frequencies or importance scores.

• Generalization: Instead of set cardinality, it uses the sum of weights.
• Formula: J_w(A, B) = Σ min(w_A(x), w_B(x)) / Σ max(w_A(x), w_B(x))
• Application: This is crucial for comparing TF-IDF vectors, where term importance varies, providing a more nuanced similarity measure for canonical content grouping.

The Jaccard Index is a primary signal in automated canonicalization workflows to detect duplicate content and near-duplicate pages.

• Duplicate Detection: A Jaccard Index of 1.0 on shingled content signals an exact duplicate, triggering a canonical tag or 301 redirect.
• Near-Duplicate Clustering: A high Jaccard Index (e.g., > 0.8) groups pages with boilerplate variations, allowing the system to select a canonical Golden Record.
• Crawl Budget Optimization: By identifying and consolidating these clusters, search engine bots avoid wasting resources on redundant URLs.