Exactly-Once Semantics

The first pillar of EOS. An idempotent producer ensures that retrying a send operation does not result in duplicate records. The broker assigns a Producer ID (PID) and sequence number to each message. If a retry sends a message with a sequence number already committed, the broker discards it as a duplicate. This prevents the common 'at-least-once' duplication scenario where a producer retries after a network timeout without knowing the original write succeeded.

The second pillar enabling end-to-end EOS across multiple partitions and topics. The transactional API allows an application to group multiple produce and consume operations into an atomic unit. Key properties:

• Atomic multi-partition writes: Either all messages across partitions are committed, or none are visible.
• Read-Process-Write: Consume from a source topic, transform, and produce to a sink topic exactly once.
• Isolation: Consumers configured with read_committed isolation level see only committed messages, filtering out aborted transactions and duplicates.

The underlying mechanism for distributed transactions in streaming systems. The process:

1. Begin Transaction: Producer registers a new transactional session.
2. Write Phase: Messages are written to partitions but marked as 'pending'—invisible to read_committed consumers.
3. Commit/Abort: The producer sends a commit marker to all involved partitions. This marker atomically flips pending messages to committed. If the producer crashes before committing, the transaction coordinator times out and aborts, preventing partial writes.

EOS must handle producer failures gracefully. Zombie fencing prevents a crashed and restarted producer from corrupting state:

• Each new transactional producer session receives a monotonically increasing epoch number.
• If a zombie producer (with an older epoch) attempts to commit or send, the broker rejects the request with a ProducerFencedException.
• The transaction coordinator uses this epoch to guarantee that only the latest active instance can finalize a transaction, ensuring correctness during failover.

EOS is not free. The guarantees impose measurable overhead:

• Increased Latency: Transactional commits require additional round-trips to the transaction coordinator.
• Reduced Throughput: Up to 20-30% throughput reduction compared to at-least-once semantics due to coordination and marker writes.
• State Overhead: Brokers must maintain transaction state and pending markers in memory.
• Best Practice: Use EOS only for critical financial, billing, or inventory pipelines where duplication is unacceptable. For analytics pipelines tolerant of rare duplicates, at-least-once with idempotent consumers is often sufficient.

Achieving true end-to-end EOS requires coordination beyond the streaming platform. When the sink is an external database or service:

• Idempotent Writes: Design the sink to accept a unique key and perform an upsert, making duplicate writes harmless.
• Two-Phase Commit Integration: Use connectors that implement a two-phase commit protocol with the external system (e.g., Kafka Connect with JDBC sink).
• Outbox Pattern: Atomically write the event and update the business entity in the same local database transaction, then stream the event. This guarantees that the event is published if and only if the state change persists.

A weaker delivery guarantee where the system ensures no data loss but may introduce duplicates during retries. If a producer doesn't receive an acknowledgment due to a network timeout, it resends the message. The consumer must therefore be idempotent—able to handle and deduplicate repeat messages. This is the default guarantee in many systems like Apache Kafka without transactions, trading strict correctness for higher throughput and lower latency.

A fire-and-forget guarantee where messages are delivered zero or one time, with no retries on failure. If an acknowledgment is lost, the message is simply gone. This provides the lowest latency and highest throughput because the system never buffers or retransmits. It's suitable for non-critical telemetry, metrics, or sensor data where occasional loss is acceptable and freshness trumps completeness.

A producer configured to ensure that retrying a send operation does not result in duplicate records in the log. In Apache Kafka, this is achieved by assigning each producer a unique ID and each message a sequence number. The broker tracks the last committed sequence per partition and rejects any retry with a lower or equal sequence, providing exactly-once semantics at the producer-to-broker boundary.

A mechanism that allows a producer to atomically write to multiple partitions or topics and a consumer to atomically commit offsets back to the broker. In Apache Kafka, transactions span both input and output operations, enabling read-process-write patterns where consuming a message, transforming it, and producing a result are treated as a single atomic unit. This is the foundation for end-to-end exactly-once semantics.

A distributed algorithm that coordinates all participants in a transaction to either commit or abort atomically. In phase one, the coordinator asks all participants to prepare; in phase two, it tells them to commit or roll back. While foundational to transactional systems, it introduces blocking behavior if the coordinator fails. Modern streaming systems often optimize this with non-blocking variants or epoch-based commit protocols.

A consumer designed to produce the same result regardless of how many times it processes the same message. Strategies include:

• Deduplication by key: Storing processed message IDs in a state store and skipping repeats.
• Upsert operations: Writing results idempotently so reprocessing overwrites rather than duplicates.
• Deterministic logic: Ensuring business logic has no side effects that compound on replay. This is essential when exactly-once semantics cannot be guaranteed by the infrastructure alone.