Log Compaction

Unlike time-based retention, log compaction retains the last known value for every unique message key. It operates by removing older records with identical keys, effectively turning the log into a key-value store of current state. This is critical for restoring stateful services after failure or scaling operations.

• Guarantees at least one record per key
• Deletes are signaled by a tombstone record (a record with a null value)
• Compaction runs as a background process on immutable segments

To explicitly delete a key from the compacted log, a producer sends a tombstone—a record with the key and a null value. The compaction process retains this tombstone for a configurable period (delete.retention.ms) before physically removing it.

• Prevents deleted keys from reappearing on partition leader changes
• Consumers must handle null values as explicit delete signals
• Tombstone retention must be longer than the maximum expected consumer downtime

Log compaction breaks the strict sequential offset guarantee. Since records with duplicate keys are removed, offsets become non-contiguous. Consumers cannot assume that offset N+1 exists just because offset N was consumed.

• Consumers must not rely on offset gaps for logic
• Re-consuming a compacted topic from offset 0 yields only the latest value per key
• Ideal for KTable state restoration in Kafka Streams

Log compaction excels for state restoration and change data capture scenarios where the current value matters more than the full history.

• Ideal: Database changelogs, Kafka Streams state stores, shipping addresses
• Anti-pattern: Immutable event logs (e.g., financial transactions, clickstreams) where every event must be retained
• Anti-pattern: Topics with infinite unique keys, as compaction cannot delete any records

Standard time-based retention deletes entire segments after a time threshold. Log compaction operates on key granularity within segments, merging non-compacted segments into compacted ones.

• Time-based: Deletes all records older than T
• Compaction: Deletes older records with the same key, regardless of age
• Both policies can be combined: a topic can have both retention.ms and cleanup.policy=compact

The Log Cleaner thread builds an in-memory offset map for each compacted segment. It scans from the latest segment backward, retaining only the highest offset for each key.

• Operates on immutable segments; never modifies the active segment
• Dirty ratio (min.cleanable.dirty.ratio) controls when cleaning triggers
• A segment is eligible when the ratio of dirty (obsolete) to total records exceeds the threshold

A persistent or in-memory storage engine used by stream processors to maintain the intermediate state required for stateful operations like aggregations and joins.

• Relationship to Compaction: A compacted topic acts as a durable, external changelog for a state store. During recovery, the store can be rebuilt by replaying the compacted topic, reading only the latest value for each key.
• RocksDB is a common local state store implementation that leverages compaction for efficient lookups.

A fault-tolerance mechanism that periodically saves the state of an operator or application to durable storage, enabling recovery from failures without data loss.

• Distinction: Checkpointing captures the entire internal state of a processing graph (including offsets and local state), while log compaction provides a changelog-based recovery path.
• In frameworks like Apache Flink, checkpoints are coordinated, globally consistent snapshots, whereas a compacted topic is an asynchronous, continuous cleanup process.

A design pattern that identifies and captures row-level changes to a source database and streams them as events to downstream systems in real-time.

• Synergy: CDC tools like Debezium often stream database changes into compacted Kafka topics. The compaction ensures the topic retains the current state of every database row, making it a perfect source for rebuilding materialized views.
• This pattern effectively turns a database's transaction log into a scalable, queryable event stream.

A software architecture pattern that treats all data as a stream, using a single stream processing engine for both real-time and batch processing.

• Role of Compaction: In a Kappa architecture, compacted topics serve as the long-term, queryable master dataset. Reprocessing historical data is done by simply replaying the compacted log, eliminating the need for a separate batch layer.
• This unifies operational and analytical data pipelines under a single streaming backbone.

A centralized service that stores and manages the schemas for data formats like Apache Avro or Protobuf, ensuring compatibility and enforcing governance in data pipelines.

• Critical Dependency: Log compaction relies on stable, consistent message keys. A schema registry enforces backward and forward compatibility on the key schema, preventing compaction failures caused by serialization mismatches.
• Without schema governance, a change to the key structure can render an entire compacted topic unrecoverable.

A pre-computed, persisted result of a query, continuously updated by a stream processor, providing low-latency read access to aggregated or joined data.

• Implementation: A materialized view is often backed by a state store that is itself populated and restored from a compacted topic. The compaction ensures the view's underlying dataset remains bounded in size.
• This pattern is fundamental to CQRS, where the query model is a materialized view built from a compacted event stream.