Idempotent Producer

To maintain the deterministic sequence number order, the idempotent producer restricts the maximum number of unacknowledged requests in flight per connection (max.in.flight.requests.per.connection) to 5 or fewer. This constraint is critical because it guarantees that the broker never sees a sequence number gap that could be misinterpreted as an out-of-order write, which would cause a fatal OutOfOrderSequenceException.

The broker uses a combination of the PID and an epoch number to fence zombie producers. When a producer with the same transactional ID reconnects after a failure, the broker bumps its epoch. Any subsequent write attempt from the old producer instance with a stale epoch is immediately rejected with a ProducerFencedException. This mechanism prevents split-brain scenarios where a presumed-dead producer comes back and corrupts the log.

Enabling idempotence introduces a minimal, constant-factor overhead. The primary cost is the broker-side validation of sequence numbers and the maintenance of the PID cache. In practice, this overhead is negligible for most workloads, adding only microseconds of latency. However, it does require acks=all and a replication factor of at least 3 to ensure the PID state is durable, which increases write latency compared to acks=1.

The delivery guarantee that an idempotent producer helps achieve. In distributed streaming, exactly-once semantics ensures that a message is both delivered and processed precisely one time, even when producers retry sends due to network failures or broker timeouts. This is distinct from at-least-once (which can produce duplicates) and at-most-once (which can lose messages). The idempotent producer addresses the producer-to-broker leg of this guarantee, preventing duplicate writes when acks=all acknowledgments are delayed but the original write actually succeeded.

The internal mechanism that enables idempotency. When enable.idempotence=true is configured, the broker assigns a unique Producer ID (PID) during initialization. The producer then tags every message batch with the PID and a monotonically increasing sequence number. The broker tracks the last committed sequence number per PID-topic-partition combination. If a retry sends a batch with a sequence number less than or equal to the last committed value, the broker silently discards it as a duplicate, returning acknowledgment without rewriting the data.

An extension of the idempotent producer that adds atomicity across multiple partitions. While an idempotent producer prevents duplicates within a single partition, a transactional producer wraps writes to multiple partitions in an atomic unit using beginTransaction() and commitTransaction() calls. This guarantees that either all messages in a transaction are written and committed, or none are visible to consumers configured with isolation.level=read_committed. The transactional producer requires a unique transactional.id and builds on the same PID and epoch mechanisms.

The failure scenario that necessitates idempotency. When retries > 0 and max.in.flight.requests.per.connection > 1, a producer can send multiple batches without waiting for acknowledgment. If a network partition delays acknowledgment of batch N but batch N+1 succeeds, a retry of batch N would normally write a duplicate. The idempotent producer resolves this by enforcing ordering guarantees: with enable.idempotence=true, max.in.flight.requests.per.connection is capped at 5, and the broker uses sequence numbers to reject out-of-order duplicates.

A mechanism to prevent zombie producers from corrupting the log. When a producer with a given transactional.id or PID reconnects after a session expiration, the broker increments its producer epoch. Any inflight or delayed requests bearing an older epoch are immediately rejected with a PRODUCER_FENCED error. This fencing token ensures that a producer instance that has been superseded cannot write stale data or commit a dangling transaction, protecting the integrity of the exactly-once guarantee across restarts and failovers.

A critical distinction in responsibility. The idempotent producer handles producer-level deduplication—preventing the same send operation from creating duplicate records in the log. This is distinct from consumer-side deduplication, where the consuming application must track processed message IDs to handle cases where the broker delivers the same message multiple times (e.g., after a consumer rebalance). True end-to-end exactly-once processing requires both: idempotent producers to prevent duplicate writes and idempotent consumers or transactional reads to handle duplicate deliveries.