Bloom Filter

A Bloom filter represents a set using a fixed-size bit array and k independent hash functions. To add an element, it is fed to each hash function, and the resulting array positions are set to 1. This structure requires significantly less memory than storing the elements themselves, making it ideal for high-volume, memory-constrained environments like network routers and database cache layers.

The core operational guarantee is asymmetric:

• Definitive Absence: If any bit position for a queried element is 0, the element has definitively never been added. There are no false negatives.
• Probabilistic Presence: If all bit positions are 1, the element may be in the set, or it may be a false positive caused by hash collisions from other inserted elements. The false positive rate is tunable based on the array size (m), number of hash functions (k), and the number of inserted elements (n).

The probability of a false positive (p) is engineered during initialization and is approximated by the formula: (1 - e^(-kn/m))^k. System designers balance three parameters:

• Bit Array Size (m): Larger arrays reduce collisions but consume more memory.
• Number of Hash Functions (k): Too few increase collisions; too many fill the array too quickly. The optimal number is k = (m/n) * ln(2).
• Expected Elements (n): The filter must be sized for an anticipated cardinality. Exceeding this count rapidly degrades accuracy.

A standard Bloom filter does not support removing elements. Resetting a 1 to a 0 could inadvertently remove other elements that hashed to the same bit, introducing false negatives and violating the core guarantee. To support deletion, a variant called a Counting Bloom Filter replaces each bit with a small counter, incrementing on insertion and decrementing on removal, at the cost of increased memory overhead.

Both insertion and query operations execute in O(k) constant time, where k is the fixed number of hash functions. This performance is independent of the number of elements already stored in the filter. This predictable, low-latency behavior is critical for real-time stream processing systems like Apache Kafka and Apache Flink, where a Bloom filter can filter billions of events per second before an expensive state lookup.

Bloom filters are a foundational component in modern data infrastructure for reducing I/O and network overhead:

• Apache Cassandra & HBase: Use them to avoid costly disk seeks for non-existent rows in SSTables.
• Content Delivery Networks (CDNs): Cache proxies use them to avoid caching one-hit-wonder web objects.
• Google Bigtable: Employs them to suppress lookups for non-existent rows and columns.
• Cryptocurrency: SPV clients use them to request relevant transactions from full nodes without revealing their entire wallet.

A probabilistic algorithm for cardinality estimation—counting the number of unique elements in a massive data stream. Like a Bloom filter, it uses a fixed, tiny memory footprint (typically a few kilobytes) regardless of the stream size.

• Achieves ~2% error rate with just 1.5 KB of memory
• Used for real-time unique visitor counting and reach estimation in ad tech
• Based on observing the longest run of leading zeros in hashed values

A probabilistic data structure for estimating the frequency of events in a data stream, such as counting how many times a user has viewed a product. It provides a frequency estimate that is always an overcount, never an undercount.

• Guarantees one-sided error: estimate ≥ true count
• Used for heavy-hitter detection and trending topic identification
• Sub-linear space complexity relative to the number of distinct items tracked

An alternative to the Bloom filter that supports dynamic deletion of elements, which standard Bloom filters cannot do without rebuilding. It uses cuckoo hashing to relocate existing entries when collisions occur.

• Supports add and remove operations natively
• Often achieves better space efficiency for low false-positive rates (< 3%)
• Used in network packet deduplication and database query optimization

A technique for quickly estimating the Jaccard similarity between two sets without comparing every element. It compresses large sets into compact signatures, enabling rapid deduplication and clustering.

• Used by search engines for near-duplicate document detection
• Forms the foundation of Locality-Sensitive Hashing (LSH)
• Enables similarity comparisons in sub-linear time

A space-efficient probabilistic data structure that, like the Bloom filter, supports approximate membership queries. Its key advantage is cache-friendly access patterns due to storing elements in contiguous memory slots.

• Offers better cache locality than standard Bloom filters
• Supports merge and resize operations efficiently
• Used in key-value stores and log-structured merge trees

The deliberate tuning of a Bloom filter's parameters—bit array size (m) and number of hash functions (k)—to achieve a target false positive probability. The optimal k is approximately (m/n) * ln(2), where n is the expected number of inserted elements.

• Doubling the bit array roughly squares the false positive rate
• Critical for real-time segmentation where memory budgets are fixed
• Trade-off: lower error rates require more memory or more hash computations