Exactly-Once Semantics

The fundamental mechanism enabling exactly-once semantics. An operation is idempotent if applying it multiple times has the same effect as applying it once.

• Producer idempotency: Assigning unique sequence numbers to each message so the broker can deduplicate retries
• Consumer idempotency: Using transactional offsets or upsert logic to ensure reprocessing a message doesn't create duplicate side effects
• Example: A command to set valve position to 75% is naturally idempotent; increment counter by 1 is not and requires additional deduplication logic

An atomic write protocol that couples message production with offset commits into a single all-or-nothing operation.

• Producers write messages and commit offsets within a transaction boundary
• If any part fails, the entire transaction is aborted and can be retried safely
• Two-phase commit protocol: A transaction coordinator manages prepare and commit phases across partitions
• This prevents the common failure mode where a consumer crashes after processing but before committing its offset, causing duplicate processing on restart

Producers assign a monotonically increasing sequence number to each message directed at a specific partition. The broker maintains the last committed sequence number per producer session.

• On receiving a message, the broker checks the sequence number against its stored value
• If the sequence number is less than or equal to the last committed value, the message is silently discarded as a duplicate
• If it is exactly last + 1, the message is appended to the log
• This guards against producer retries caused by transient network failures where the original write succeeded but the acknowledgment was lost

A guarantee that messages written to multiple partitions or topics within a single transaction are either all visible or none visible to consumers.

• Consumers configured with isolation.level=read_committed will only see messages from committed transactions
• Messages from aborted or in-flight transactions are filtered out, preventing consumers from acting on partial or rolled-back data
• This is critical for industrial workflows where a single logical event—like a recipe change—must atomically update parameters across multiple equipment partitions

A mechanism to prevent zombie processes—producers or consumers that were partitioned from the system but believe they are still active—from corrupting state.

• Each producer is assigned an epoch (or generation ID) when it registers with the broker
• Older epochs are fenced off: any write attempt with an expired epoch is rejected with a fatal error
• This guarantees that only the most recent incarnation of a producer can write, preventing split-brain scenarios where two instances of the same logical producer emit conflicting messages

Exactly-once systems define precise recovery behavior for each failure mode to maintain the guarantee.

• Producer failure: On restart, the producer initializes with a fresh epoch. In-flight transactions are aborted by the broker after a timeout
• Broker failure: Transaction state is replicated to followers. On leader failover, the new leader reconstructs transaction metadata from the replicated log
• Consumer failure: When a consumer in a group fails, partition reassignment triggers a rebalance. The new consumer reads committed offsets and resumes processing without gaps or duplicates
• Network partition: Producers receive a timeout and retry with a new sequence number; the broker's deduplication layer handles any duplicate writes that made it through

A weaker delivery guarantee where the system ensures no data loss but may introduce duplicates. If a producer does not receive an acknowledgment, it retransmits the message. The consumer must therefore be idempotent—able to safely handle and deduplicate repeated records. This is the default guarantee in many systems like Apache Kafka (default producer settings) and is simpler to implement than exactly-once because it avoids the overhead of distributed transactions.

The simplest delivery guarantee where messages are sent without retries. If a transmission fails, the message is lost and not resent. This is suitable for non-critical telemetry where freshness trumps completeness, such as high-frequency ambient temperature readings where a single missed data point is statistically insignificant. Systems like UDP-based sensor broadcasts often operate on this model to minimize latency and overhead.

A foundational building block for exactly-once semantics. An idempotent producer assigns a unique sequence number to each message. The broker tracks these sequences and deduplicates any retransmitted messages with the same identifier. In Apache Kafka, enabling enable.idempotence=true ensures that retries caused by transient network failures do not create duplicate records within a single partition, preventing the producer from being a source of duplication.

A design pattern critical for achieving exactly-once processing across heterogeneous systems (e.g., a database and a message broker). Instead of attempting a distributed transaction, the service writes the outgoing message to an outbox table within the same local database transaction as the business logic. A separate relay process then reads the outbox and publishes to the message broker, ensuring atomicity: either both the state change and the message are persisted, or neither is.

A distributed consensus protocol used by some stream processors to coordinate a transaction across multiple partitions or external sinks. In the first prepare phase, all participants agree to commit. In the second commit phase, the transaction is finalized. Apache Flink uses a variant of 2PC with distributed snapshots (checkpoints) to implement end-to-end exactly-once state consistency, ensuring that state updates and output commits happen atomically at checkpoint boundaries.

A specialized queue that acts as a safety valve for messages that cannot be processed successfully after all retries are exhausted. In an exactly-once pipeline, a poison-pill message—one that causes deterministic processing failures—can block the entire stream. Routing such messages to a DLQ allows the main pipeline to continue processing healthy records while enabling manual inspection and remediation of the failed messages without violating delivery guarantees.