Topic

A topic is divided into partitions, which are ordered, immutable sequences of records. Each partition is a unit of parallelism—multiple consumers can read from different partitions simultaneously, enabling horizontal scalability. Partitions are distributed across brokers in a cluster, and the number of partitions determines the maximum parallelism of consumption. Messages within a partition maintain strict total order, but ordering across partitions is not guaranteed.

A topic is implemented as an append-only commit log stored on disk. Once a message is written, it is immutable—it cannot be altered or deleted by normal operations. This design provides durability and enables features like replayability and auditability. Retention policies control how long messages persist, configurable by time (e.g., 7 days) or size (e.g., 1 TB). Log compaction is a specialized retention strategy that retains only the latest value per key, useful for maintaining current state.

Producers publish messages to a topic without knowledge of the consumers. Subscribers express interest by subscribing to the topic and consume messages at their own pace. This temporal decoupling means producers and consumers do not need to be online simultaneously. Spatial decoupling means they do not need to know each other's network addresses. This loose coupling is foundational to event-driven architectures and microservices communication.

Consumers are organized into consumer groups for load-balanced consumption. Each partition is consumed by exactly one member within a group, enabling parallel processing. Consumers track their position using an offset—a unique sequential ID per record. Offsets are committed periodically, allowing consumers to resume from their last committed position after failure. This model supports at-least-once or exactly-once delivery semantics depending on configuration.

Messages in a topic are typically serialized with a schema-based format like Apache Avro or Protobuf. A Schema Registry stores and validates schemas, enforcing compatibility rules (backward, forward, full) as topics evolve. This prevents data corruption and ensures that producers and consumers agree on the structure of messages. Schema evolution is critical for maintaining data contracts in long-lived streaming pipelines.

Each partition is replicated across multiple brokers for high availability. One broker acts as the leader for a partition, handling all reads and writes, while followers passively replicate the data. If the leader fails, a follower is automatically elected as the new leader. The replication factor (typically 3) determines the number of copies, balancing durability against storage cost. This design ensures zero data loss in the face of hardware failures.

A computation engine that continuously processes unbounded streams of data records to perform operations like filtering, aggregation, and joining in real-time. Unlike batch processing, stream processors handle data as it arrives.

• Examples include Apache Flink, Kafka Streams, and Spark Streaming
• Supports stateful operations using embedded state stores
• Uses windowing to divide infinite streams into finite chunks for analysis

A delivery guarantee ensuring that a message is processed only once, even in the face of network failures or producer retries. This prevents both data duplication and data loss in critical pipelines.

• Achieved through idempotent producers and transactional coordination
• Requires cooperation between the messaging system and the stream processor
• Essential for financial transactions and inventory management systems

A design pattern that identifies and captures row-level changes to a source database and streams them as events to downstream systems in real-time. CDC eliminates the need for batch ETL by turning database commits into event streams.

• Tools like Debezium tail database transaction logs
• Enables real-time cache invalidation and search index updates
• Preserves the full history of mutations for audit and replay

A technique that divides an unbounded data stream into finite chunks based on temporal boundaries for performing aggregations like counts, sums, or averages. Windows define the scope of a computation over time.

• Tumbling windows: Fixed-size, non-overlapping intervals
• Sliding windows: Overlapping intervals that slide by a step size
• Session windows: Dynamic windows based on periods of user inactivity