Change Data Capture (CDC)

CDC identifies and captures only the inserts, updates, and deletes that occur in a source database, rather than performing full-table scans. This is achieved by monitoring the database's transaction log (e.g., Write-Ahead Log in PostgreSQL, binary log in MySQL, or redo log in Oracle).

• Efficiency: Processes only changed data, drastically reducing network and compute load.
• Low Latency: Enables near real-time data propagation, often with sub-second latency.
• Minimal Source Impact: Avoids expensive SELECT queries on source tables, preserving performance for operational workloads.

CDC transforms database changes into a continuous, ordered stream of change events. This stream serves as the foundation for event-driven architectures and real-time analytics.

• Event Serialization: Changes are typically emitted as structured messages (e.g., JSON, Avro, Protocol Buffers) containing the operation type, before/after state, and metadata like transaction ID and timestamp.
• Ordering Guarantees: Maintains commit-order consistency, ensuring downstream consumers see changes in the same sequence they occurred at the source.
• Integration: Feeds directly into stream-processing platforms like Apache Kafka, Amazon Kinesis, or Google Pub/Sub for further transformation and routing.

CDC systems maintain durable, fault-tolerant state to guarantee exactly-once or at-least-once delivery semantics, even during failures.

• Offset/Checkpoint Management: The CDC process persistently records its position in the source log (e.g., LSN - Log Sequence Number). After a restart, it resumes from the last committed offset, preventing data loss.
• Idempotent Sinks: When integrated with systems that support idempotent writes, CDC can ensure each change is applied exactly once to the target, crucial for financial or inventory data.
• Debezium is a prominent open-source CDC platform that implements these stateful connectors for various databases.

As source database schemas change (e.g., adding a column), CDC systems must adapt without breaking downstream pipelines. This involves capturing and propagating schema metadata alongside the data.

• Schema Registry Integration: Tools like the Confluent Schema Registry store Avro, JSON Schema, or Protobuf schemas, allowing consumers to deserialize messages correctly across versions.
• Backward/Forward Compatibility: CDC events are often structured to be compatible with older and newer consumer versions, using techniques like setting missing fields to null.
• Snapshotting: On connector start or after a schema change, a CDC system may take a consistent snapshot of the current table state to re-baseline the stream.

CDC is the primary method for synchronizing data across disparate systems with different data models, query languages, and performance characteristics.

• Use Cases:
  • Data Warehouse/Lakehouse Ingestion: Streaming changes from OLTP databases into analytical platforms like Snowflake, Databricks, or Google BigQuery.
  • Search Index Updates: Populating Elasticsearch or OpenSearch indices in real-time for fresh search results.
  • Cache Invalidation: Updating Redis or Memcached caches when underlying database records change.
  • Microservices Data Sharing: Propagating state changes between bounded contexts in a decoupled manner.

A complete CDC deployment must handle not only ongoing changes but also the initial population of target systems with existing historical data.

• Process: The connector first performs a consistent snapshot of the source tables. This can be a blocking read with locks, a non-blocking read using transaction isolation, or by leveraging a previously taken database backup.
• Stream Continuity: After the snapshot is complete, the connector seamlessly transitions to reading the transaction log from the point corresponding to the snapshot's consistency point, ensuring no data is missed or duplicated.
• Performance: For large tables, snapshots are performed in chunks to avoid overwhelming the source database or exhausting the connector's memory.

A software architecture pattern where the state of an application is determined by a sequence of immutable events, which are stored as the system's single source of truth. Instead of storing the current state, the system persists the history of state-changing actions.

• Core Principle: The current state is a derivative of the event log; rebuilding state involves replaying all events.
• Relation to CDC: While CDC captures changes from an existing database (a system of record), Event Sourcing designs the system around the change log from the start. CDC can be used to propagate events from an Event-Sourced system to other services.

A fundamental database protocol that ensures data durability and supports transaction atomicity. All modifications (inserts, updates, deletes) are first written to a persistent, append-only log file before they are applied to the main database files (data pages).

• Mechanism: The WAL acts as a sequential record of intended changes. In a crash, the database can recover by "replaying" the log.
• Relation to CDC: The database transaction log (often implemented as a WAL) is the primary data source for many CDC implementations (e.g., PostgreSQL logical decoding, MySQL binlog). CDC tools essentially stream and interpret the WAL.

The process of copying and maintaining database objects (tables, rows) in multiple distinct locations to improve availability, reliability, and fault tolerance. It is a broader objective for which CDC is a key enabling technology.

• Types: Includes snapshot replication, merge replication, and transactional replication.
• CDC's Role: CDC is the engine for real-time transactional replication. It captures incremental changes at the source and applies them with low latency to one or more replicas, enabling use cases like read scaling, geo-distribution, and hot standby failover.

The maintenance and assurance of the accuracy and consistency of data over its entire lifecycle. It encompasses protection from corruption, unauthorized alteration, and ensuring data remains an accurate reflection of the real-world entities it represents.

• Critical Aspects: Includes entity integrity, referential integrity, and transactional integrity (ACID properties).
• Relation to CDC: A robust CDC system must preserve data integrity during the capture and propagation process. This means ensuring exactly-once semantics (or at-least-once with idempotent consumers), maintaining transactional ordering where required, and applying changes without violating constraints at the target.