Apache Kafka

Kafka is fundamentally a distributed, partitioned, replicated commit log service. Data is written to append-only logs called topics, which are partitioned across multiple brokers. This architecture provides:

• O(1) disk access: Sequential I/O eliminates random disk seeks, enabling extremely high throughput on commodity hardware.
• Immutable records: Once written, records cannot be modified, ensuring a durable, ordered, and replayable sequence of events.
• Message retention: Unlike traditional message queues, Kafka retains messages for a configurable period (e.g., 7 days) regardless of consumption, allowing new consumers to reprocess historical data.

Kafka decouples data producers and consumers through a pub/sub pattern centered on topics. This creates a highly scalable, loosely coupled data bus:

• Producers: Client applications that publish records to specific topics, optionally specifying a partition key for ordering guarantees.
• Consumers: Applications that subscribe to one or more topics and pull data at their own pace, tracked by an offset pointer.
• Consumer Groups: A set of consumers that cooperatively consume from a topic. Each partition is assigned to exactly one consumer within a group, enabling horizontal scaling of processing while maintaining strict ordering per partition.

Kafka achieves high availability and durability through leader-follower replication of topic partitions:

• Each partition has one leader broker that handles all reads and writes, and zero or more follower brokers that passively replicate the log.
• In-Sync Replicas (ISR): The set of replicas that are fully caught up with the leader. Only ISR members are eligible for leader election.
• Automatic failover: If a leader fails, a new leader is automatically elected from the ISR set by the Kafka controller, with zero data loss when using acks=all producer configuration.
• This design allows Kafka clusters to survive broker failures without manual intervention, making it suitable for mission-critical data pipelines.

Kafka includes a native stream processing library that transforms data directly within the Kafka ecosystem, eliminating the need for separate processing clusters:

• Stateful operations: Supports windowed aggregations, joins, and interactive queries over local state stores backed by RocksDB.
• Exactly-once semantics: Guarantees that each record is processed exactly once, even in the face of failures, using idempotent producers and transactional messaging.
• KTable abstraction: Represents a changelog stream where each record is an upsert, enabling real-time materialized views of data.
• Embeddable library: Runs directly within any Java application, with no separate processing cluster required, simplifying deployment and operations.

Kafka provides strong delivery guarantees through configurable producer and consumer semantics:

• At-most-once: Messages may be lost but never duplicated (default, low overhead).
• At-least-once: Messages are never lost but may be duplicated (achieved with retries and acks=all).
• Exactly-once: Messages are delivered and processed exactly once, end-to-end, using:
  • Idempotent producers: Detect and eliminate duplicate writes using producer IDs and sequence numbers.
  • Transactional API: Atomically write to multiple partitions and commit consumer offsets, ensuring read-process-write cycles are atomic.
• This is critical for financial systems, inventory management, and any use case where duplicate processing would corrupt state.

Beyond time-based retention, Kafka offers log compaction to maintain the latest state for each key:

• Instead of deleting old segments by age, compaction retains only the most recent value for each unique key within a topic.
• Null value tombstoning: Writing a record with a null value signals deletion, and the key is eventually removed after a configurable retention period.
• This enables Kafka to function as a distributed key-value store or a change data capture (CDC) source, where the compacted topic represents a snapshot of the current state of a database table.
• Compaction runs asynchronously in the background on a dedicated thread pool, ensuring it does not impact read/write performance.

A stream processing operation that continuously computes summary statistics—such as sum, average, or count—over a finite, time-bounded subset of an infinite event stream. Windows can be tumbling, sliding, or session-based.

• Tumbling windows: Fixed-size, non-overlapping intervals (e.g., every 5 minutes)
• Sliding windows: Overlapping intervals for smoother trend analysis
• Session windows: Dynamic boundaries based on periods of user inactivity
• Use case: Calculating a user's page views in the last 30 seconds to adjust content ranking

A mechanism in stream processing that tracks the progress of event time and provides a threshold for tolerating late-arriving data. A watermark declares that no events with a timestamp older than the watermark are expected to arrive.

• Purpose: Balances result completeness with processing latency
• Trade-off: A longer wait yields more complete results but increases output delay
• Use case: Waiting up to 3 seconds for out-of-order click events before finalizing a session window

A delivery guarantee in stream processing ensuring that even in the event of failures, each record is processed only once, and the resulting application state is computed as if the failure never occurred. This prevents duplicate counting or missed updates.

• Implementation: Achieved through idempotent producers and transactional sinks
• Critical for: Financial calculations, inventory counts, and loyalty point accruals
• Use case: Ensuring a user's real-time discount counter is not double-incremented during a broker failover