Exactly-Once Delivery

The application logic on the receiving service must be idempotent, meaning processing the same message multiple times produces the same final state as processing it once. This is the foundational defense against duplicates that slip through protocol guarantees.

• Implementation: Use a unique idempotency key (e.g., derived from the message) to track processed operations in a persistent store.
• Check-Before-Process: Before acting, the consumer verifies the key hasn't been seen before.
• Example: A payment service receiving a "deduct $10" command must ensure that duplicate commands do not result in $20 being deducted.

This pattern ensures atomicity between publishing a message and committing the business transaction that generated it, preventing a scenario where one succeeds and the other fails (causing loss).

• Mechanism: The application writes both the business data and the outgoing message to the same database transaction, storing the message in an "outbox" table.
• Relay Process: A separate process (publisher) reads from the outbox table and publishes to the message broker, then marks the message as sent.
• Guarantee: The message is guaranteed to be published if and only if the business transaction is committed.

The message broker itself can provide exactly-once semantics by tracking producer and message identifiers to filter duplicates before they reach consumers.

• Producer IDs & Sequence Numbers: The broker assigns each producer a unique ID. The producer attaches a monotonically increasing sequence number to each message within a partition/topic.
• Broker Logic: The broker rejects any message whose sequence number is not greater than the last committed number for that producer-partition pair.
• System Examples: Apache Kafka (with enable.idempotence=true) and Apache Pulsar implement this broker-level deduplication, providing exactly-once delivery in the write path.

For systems where processing a message results in publishing new messages or writing to a database, transactional consumption coordinates the input consumption, processing, and output publishing as a single atomic unit.

• Mechanism: The consumer reads a batch of messages but does not commit its offset (acknowledge receipt) until the entire processing cycle—including any resulting database writes and outgoing messages—is complete.
• Failure Handling: If any part fails, the transaction aborts, the consumer's offset is not advanced, and the messages are re-fetched for reprocessing.
• Framework Support: This is a core feature of stream processing frameworks like Apache Kafka Streams and Apache Flink, which manage distributed state snapshots to enable fault-tolerant, exactly-once stream processing.

In advanced stream processing, achieving end-to-end exactly-once across multiple stateful processing stages requires distributed consensus on system state and periodic checkpointing.

• Checkpointing: The framework periodically takes a consistent snapshot of all operator states and input offsets, persisting it to durable storage (e.g., HDFS, S3).
• Recovery: Upon failure, the system rolls back to the last completed checkpoint, resets input sources to those recorded offsets, and recomputes the lost work.
• Chandy-Lamport Algorithm: A foundational algorithm for creating globally consistent snapshots in distributed dataflows without stopping the system. This ensures all parts of the pipeline agree on a recovery point.

Exactly-once delivery is not free; it introduces latency and resource overhead that must be balanced against application requirements.

• Overhead Sources: Additional network round-trips for transaction coordination, persistent storage for deduplication logs and checkpoints, and increased CPU usage for idempotency checks.
• When to Use: Essential for financial transactions, inventory management, or any operation where duplicate or lost data has significant business cost.
• Alternatives: At-least-once delivery (with idempotent consumers) is often sufficient and more performant. At-most-once delivery offers the lowest latency but risks message loss.

Quality of Service levels are standardized delivery guarantees in messaging protocols. They define the contract between sender and receiver:

• QoS 0 (At-Most-Once): Fastest, but messages may be lost. No retries.
• QoS 1 (At-Least-Once): Guarantees delivery via acknowledgments and retries, but may cause duplicates.
• QoS 2 (Exactly-Once): Highest assurance, ensuring no loss and no duplication, but requires multi-phase commit protocols, adding latency and complexity. Protocols like MQTT implement these levels directly.

An idempotency key is a client-generated unique identifier (UUID) attached to a request, enabling the receiver to safely process the same message multiple times. This is a fundamental technique for achieving exactly-once semantics at the application level.

• The server stores the key with the result of the first successful execution.
• Subsequent requests with the same key return the stored result without re-executing the operation.
• This prevents duplicate side effects from retried messages, a common scenario in at-least-once delivery systems.

The publish-subscribe pattern is a messaging paradigm where senders (publishers) categorize messages into topics without knowledge of the receivers (subscribers). This decoupling is central to fleet orchestration.

• Agents publish status updates (e.g., robot/123/battery) to a broker.
• The orchestration middleware subscribes to relevant topics to maintain fleet state.
• Exactly-once guarantees in this pattern are complex, as the broker must ensure each subscriber gets the message once, even if there are multiple subscribers or the subscriber crashes and reconnects.

A message broker is intermediary middleware (e.g., Apache Kafka, RabbitMQ, AWS SQS) that facilitates communication between applications. It is the core component responsible for implementing delivery guarantees.

• Responsibilities: Message validation, transformation, routing, and persistence.
• Exactly-Once Implementation: Brokers like Apache Kafka achieve this through a combination of idempotent producers (using sequence numbers) and transactional writes across partitions.
• It acts as the central nervous system for inter-agent communication protocols, ensuring commands and telemetry are reliably exchanged.

The Saga pattern is a design pattern for managing long-running, distributed transactions—common in fleet operations like a multi-robot fetch-and-deliver task. It complements exactly-once messaging for business process consistency.

• A saga breaks a transaction into a sequence of local transactions, each updating a single service/agent.
• If a step fails, compensating transactions (e.g., CancelNavigation()) are executed to rollback previous steps.
• While exactly-once delivery ensures a command is received once, the saga pattern ensures the business logic across multiple agents is eventually consistent.

A Dead Letter Queue is a holding queue for messages that cannot be delivered or processed after repeated failures. It is a critical observability and recovery tool in systems aiming for exactly-once semantics.

• Causes for routing to DLQ: Poison messages (malformed data), persistent consumer failures, or processing timeouts.
• Role in Reliability: By isolating faulty messages, the DLQ prevents system-wide blockage and allows for manual or automated analysis and reprocessing.
• It acts as a safety net, ensuring the main message flow maintains its delivery guarantees while exceptions are handled separately.