Apache Kafka

At its core, Kafka is a distributed, append-only log. Producers write records to the end of a log, and consumers read sequentially using an offset pointer. This immutable, ordered structure provides:

• Total ordering within a partition
• Replayability of entire event streams from any point in time
• Durability through disk persistence, not in-memory queuing

Unlike traditional message queues that discard messages after consumption, Kafka retains data based on configurable retention policies, allowing new consumers to backfill historical data without impacting producers.

Each topic is split into multiple partitions distributed across the cluster. Partitions are the unit of parallelism:

• A topic with 100 partitions can be consumed by up to 100 consumer instances in a consumer group
• Producers can route messages to specific partitions by key, guaranteeing order for all events sharing that key
• Partitions enable horizontal scaling — adding brokers increases both throughput and storage capacity linearly

This design avoids the bottleneck of a single ordered queue while preserving ordering guarantees where they matter.

Kafka distributes consumption across a consumer group by assigning each partition to exactly one consumer instance within the group. This provides:

• Fault tolerance: If a consumer fails, its partitions are automatically reassigned to remaining members via a group coordinator
• Scale-out consumption: Adding consumers increases processing throughput up to the partition count
• Multi-subscriber fan-out: Different consumer groups read the same topic independently, each maintaining its own offset position

This model decouples production from consumption, allowing multiple downstream applications to process the same event stream at their own pace.

Every partition is replicated across a configurable number of brokers (typically 3). One broker acts as the leader for reads and writes, while followers passively replicate the log. Key guarantees:

• ISR (In-Sync Replicas): Only replicas that are fully caught up are eligible for leader election
• Acks configuration: Producers can require acknowledgment from all in-sync replicas (acks=all) for guaranteed durability
• Rack awareness: Replicas can be placed across availability zones to survive data center outages

A failed leader triggers an automatic election among ISRs, with zero data loss when acks=all and min.insync.replicas are properly configured.

Beyond time-based retention, Kafka supports log compaction — a retention mechanism that keeps only the latest record for each key. This transforms Kafka into a distributed key-value store:

• Change Data Capture (CDC): Database changes streamed into Kafka maintain a compacted snapshot of current row state
• KTable in Kafka Streams: Compacted topics back stateful stream processing operations
• Event sourcing snapshots: Aggregate state can be rebuilt from compacted topics without replaying the entire event history

Compaction runs asynchronously in the background, periodically removing obsolete records while preserving the most recent value per key.

Kafka provides exactly-once semantics (EOS) across producers, brokers, and stream processors, eliminating duplicate processing even during failures. This is achieved through:

• Idempotent producers: Each message gets a producer ID and sequence number; brokers deduplicate retries
• Transactional writes: Producers can atomically write to multiple partitions using a two-phase commit protocol
• Kafka Streams EOS: State stores and output topics are updated transactionally, preventing partial updates

This guarantee is critical for financial ledgers, inventory systems, and any use case where double-counting is unacceptable.