Change Data Capture (CDC)

CDC systems are designed for minimal latency, propagating data changes from a source (like a central fleet database) to downstream consumers (like individual agent controllers) in near real-time. This is achieved by continuously monitoring database transaction logs rather than polling tables at intervals.

• Mechanism: Tails the database's write-ahead log (WAL) or uses triggers to capture inserts, updates, and deletes as they occur.
• Benefit: Enables sub-second reaction times for agents receiving updated task assignments, zone permissions, or fleet state changes, which is critical for dynamic environments.

Reliable CDC implementations provide strong delivery guarantees to prevent data loss or duplication, which could cause agents to miss instructions or perform duplicate work.

• Exactly-Once Delivery: Ensures each change event is processed precisely one time by the consuming system, often using idempotency keys and transactional outbox patterns.
• At-Least-Once Delivery: Guarantees no data loss but may require the consumer to handle deduplication.
• Importance: In fleet orchestration, a lost 'stop' command or a duplicated 'move to charging station' event can lead to operational failures or deadlocks.

As the data models for agents and tasks evolve (e.g., adding a new sensor data field), CDC systems must handle backward and forward compatibility without breaking downstream consumers.

• Techniques: Use of serialization formats like Protocol Buffers (Protobuf) or Avro that support schema versioning and validation.
• Process: Changes to the source database schema are propagated as updated event structures, allowing older agent software to ignore new fields and newer software to leverage them.
• Critical For: Long-lived deployments where fleet hardware and software may be updated asynchronously.

A well-architected CDC solution imposes negligible performance overhead on the source database, which is often a critical transactional system for warehouse management or order processing.

• Log-Based vs. Query-Based: Log-based CDC (reading the WAL) is preferred as it does not add SELECT load to production tables, unlike trigger-based or timestamp-polling methods.
• Benefit: The operational database maintains its performance for primary transactions, while the CDC stream provides a dedicated feed for the orchestration layer and agent updates.

CDC captures not just the new state of a record, but the complete change event, which is essential for audit trails, debugging, and complex state reconstruction.

• Before/After Images: Captures the state of the data before and after the change.
• Operation Type: Identifies the change as an INSERT, UPDATE, DELETE, or TOMBSTONE (for soft deletes).
• Use Case: If an agent's assigned task is canceled, the orchestration system needs the 'DELETE' event to trigger real-time replanning and notify the agent, not just infer the change from a missing record.

CDC events are naturally consumed by stream processing frameworks (e.g., Apache Kafka, Apache Flink) that power the orchestration engine's decision-making.

• Pattern: Database changes are published to a message broker or event stream. The orchestration middleware subscribes, applies business logic (e.g., task allocation algorithms), and publishes new commands to agent-specific topics.
• Example: A warehouse_inventory table update (CDC event) → stream processor calculates needed restock → creates a pick_and_transport task (new event) → assigned to an available Autonomous Mobile Robot via its command channel.

An architectural pattern where the state of an application is determined by a sequence of immutable events, which are stored as the system's single source of truth. CDC is often used to capture these state changes from a traditional database to feed an event-sourced system.

• Core Principle: State is a derivative of an append-only log of events.
• Relation to CDC: CDC streams from an operational database can be transformed into the domain events that constitute the event log.
• Example: A warehouse inventory system stores events like ItemPicked and ItemRestocked. CDC on the inventory table generates the stream from which these events are derived.

An intermediary software module that translates messages between formal messaging protocols, enabling applications and services to communicate. CDC systems frequently publish change events to a broker for distribution.

• Function: Implements core patterns like publish-subscribe and point-to-point queuing.
• CDC Integration: Acts as the distribution backbone for CDC event streams, decoupling the source database from consuming services.
• Common Brokers: Apache Kafka, RabbitMQ, Amazon Kinesis, and Google Pub/Sub are often used as targets for CDC connectors.

A durability guarantee where modifications to data are first written to a persistent log before the actual data structures (like tables) are updated. This is the primary mechanism many CDC tools use to capture changes.

• How CDC Uses It: CDC agents read the database's transaction log (e.g., PostgreSQL's WAL, MySQL's binlog) to identify inserts, updates, and deletes in near real-time.
• Advantage: Provides a low-overhead, non-intrusive method for change capture without polling tables.
• Critical for: Ensuring data integrity and providing the ordered sequence of changes required for accurate replication.

A messaging guarantee that ensures each CDC event is delivered to the intended recipient precisely one time, without duplication or loss, despite potential network or system failures.

• Challenge: Network retries and reprocessing can lead to duplicate events, causing data corruption.
• Implementation: Often achieved using idempotency keys or transactional outbox patterns alongside CDC.
• Importance: Essential for maintaining strong consistency in downstream systems that process financial transactions or maintain aggregate counts.

A design pattern for managing data consistency in distributed transactions by breaking the transaction into a sequence of local transactions, each with a compensating transaction for rollback. CDC can be used to trigger saga steps or observe their completion.

• Use Case: In fleet orchestration, a 'Move Item' transaction might involve a robot (agent) and an inventory system.
• CDC's Role: CDC events from the inventory database can signal the completion of a local transaction, triggering the next step in the saga or initiating a compensation if a failure is detected.
• Outcome: Enables reliable, long-running business processes across heterogeneous services.

A middleware protocol and API standard for data-centric connectivity, enabling scalable, real-time, dependable, high-performance data exchange between publishers and subscribers. It represents a complementary, often more integrated, approach to data synchronization compared to database-centric CDC.

• Key Difference: While CDC extracts changes from a database log, DDS manages a global data space where applications directly publish and subscribe to data objects.
• Application: In real-time systems like robotics, DDS is used for direct agent-to-agent state sharing (e.g., pose, sensor data), whereas CDC might synchronize the fleet's operational data with a central planning database.
• Standard: Maintained by the Object Management Group (OMG).