Change Data Capture (CDC)

The most efficient CDC method reads directly from the database transaction log. This approach captures INSERT, UPDATE, and DELETE operations with minimal latency and zero impact on source application performance. Unlike trigger-based methods, log-based capture does not require schema changes to the source tables.

CDC operates on a push model, streaming changes to downstream systems as they happen. This contrasts with batch-oriented pull models. The immutable, ordered stream of events is typically published to a distributed log like Apache Kafka, allowing multiple independent consumers to subscribe to the same feed.

CDC pipelines must handle schema evolution gracefully. As source tables add or modify columns, the capture process and downstream consumers must adapt without data loss. A Schema Registry is critical for managing compatibility and ensuring the structural integrity of the change stream over time.

Enterprise-grade CDC guarantees idempotent processing. This ensures that in the event of a network failure or crash, replaying the log does not introduce duplicates. Achieving exactly-once semantics requires tight coordination between the capture agent and the stream processor to manage checkpoints and offsets.

To enable complex downstream reconciliation, CDC often captures the before-image (the row state prior to the mutation) alongside the after-image. This is essential for auditing, conflict resolution in bi-directional replication, and training machine learning models on the delta of a change.

CDC is not just a raw firehose. Mature implementations allow for content-based filtering and routing. For example, a single capture stream can be partitioned so that only changes to high-value customer tables are routed to the real-time personalization engine, while all other changes go to the data lake.

An architectural pattern where the state of a business entity is determined by an immutable, append-only sequence of all state-changing events, rather than just storing the current state. CDC is often the mechanism that publishes these events from a transactional database into an event log. The database transaction log becomes the source of truth, and the CDC connector translates row-level mutations into domain events.

A delivery guarantee ensuring that even in the event of failures, each record is processed only once, and the resulting state is computed as if the failure never occurred. CDC pipelines must coordinate idempotent writes and transactional sinks to prevent duplicate records when a connector replays a batch after a crash. Frameworks like Apache Flink implement this via distributed checkpointing.

A centralized service that stores and manages the schemas for data formats like Avro or Protobuf. In a CDC pipeline, the schema of the source table is embedded in each event. A schema registry enforces compatibility rules—such as backward compatibility—ensuring downstream consumers can deserialize events even as the source table evolves.

A mechanism in stream processing that tracks the progress of event time and provides a threshold for tolerating late-arriving data. When consuming CDC streams, watermarks allow windowed aggregations to close correctly, even if events from the source database arrive out of order due to network lag or distributed commits.

A transactional messaging pattern where a service writes to both its business tables and an 'outbox' table within a single database transaction. A CDC connector then tails the outbox table to reliably publish messages to a message broker. This guarantees at-least-once delivery without the risk of dual-write inconsistencies.

A computing paradigm that continuously processes and analyzes streams of event data in real-time. CDC is the ingestion backbone for ESP; it converts static database snapshots into a continuous, ordered stream of immutable events that stream processors can join, filter, and aggregate to detect patterns and trigger automated actions.