Change Data Capture (CDC)

The most efficient CDC method reads directly from the database's transaction log (e.g., MySQL binlog, PostgreSQL WAL, SQL Server T-Log). This approach captures row-level changes with minimal latency and negligible impact on source database performance, as it avoids polling the database with expensive SELECT queries. The log is an immutable, sequential record of all commits, making it the definitive source of truth for changes.

CDC captures the atomic state change of an individual row, not just the fact that a table was modified. Each event contains the full before-image and after-image of the row, along with the specific operation type:

• INSERT: Contains the new row data.
• UPDATE: Contains both the old and new row states.
• DELETE: Contains the row data that was removed. This granularity enables precise downstream replication and auditing.

Unlike traditional ETL which pulls data on a fixed schedule, CDC operates on a push-based, event-driven model. As soon as a transaction is committed to the source database log, the CDC connector immediately publishes the change event to a message broker like Apache Kafka or Amazon Kinesis. This enables true real-time streaming pipelines where downstream consumers react instantly to data mutations.

CDC preserves the total order of changes as they occurred on the source system. This is critical for maintaining state consistency in downstream replicas. If a row is updated twice, the CDC stream guarantees the first update is processed before the second. This sequential ordering is derived directly from the transaction log's commit sequence and is essential for exactly-once semantics in stream processing.

Robust CDC pipelines integrate with a Schema Registry to manage structural changes to the source tables over time. When a new column is added via a ALTER TABLE statement, the CDC connector captures the schema change event. Downstream consumers can then use schema compatibility rules (BACKWARD, FORWARD, FULL) to evolve their own processing logic without breaking, ensuring pipeline resilience.

CDC is not just for new changes; it provides a mechanism to bootstrap a consistent baseline state. When a connector starts, it takes a point-in-time snapshot of the existing table data and streams it as read events. Once the snapshot completes, it seamlessly transitions to streaming real-time changes from the exact log position where the snapshot finished, ensuring no data loss or duplication.

The computational engine that consumes CDC event streams. Unlike batch processing, stream processors continuously apply transformations, aggregations, and joins on data as it arrives.

• Frameworks: Apache Kafka Streams, Apache Flink, RisingWave
• Function: Enriches raw CDC events with business context before landing in sinks
• State Management: Maintains in-memory state for windowed aggregations and temporal joins

Without stream processing, CDC events are just raw changelogs. The stream processor turns them into actionable, contextualized data products.

A delivery guarantee ensuring each CDC event is processed precisely one time, eliminating duplicates and data loss. This is critical for financial and industrial transactions.

• Idempotent Producers: Assign unique sequence numbers to prevent duplicate writes during retries
• Transactional Writes: Atomically commit source offset and sink output together
• Challenges: Requires tight coordination between the CDC connector, broker, and sink; adds latency overhead

Without exactly-once semantics, a network blip during a debit transaction could double-post or vanish entirely.

A quarantine zone for poison messages—CDC events that fail processing after all retry attempts are exhausted. The DLQ prevents a single malformed record from blocking the entire pipeline.

• Failure Metadata: Each DLQ record includes the original payload, error stack trace, and failure timestamp
• Reprocessing Path: Operators can fix the root cause and replay messages from the DLQ back into the main pipeline
• Monitoring: DLQ depth is a key observability metric; a growing queue signals a systemic schema or logic error

In manufacturing, a DLQ might catch a sensor reading with an unexpected data type, preventing it from crashing a real-time quality dashboard.

A foundational concept: a stream is a changelog of events, and a table is the materialized state at a point in time. CDC bridges these two worlds.

• Stream as Source of Truth: The CDC stream captures every INSERT, UPDATE, and DELETE as an immutable event
• Table as Materialized View: Aggregating the stream produces the current state—e.g., the latest inventory count
• Bidirectional Sync: Changes to the table can be published back to the stream, enabling polyglot persistence

This duality allows a single CDC pipeline to feed both real-time dashboards and point-in-time audit reports from the same source.

A flow-control mechanism that prevents a fast CDC producer from overwhelming a slow consumer. Without backpressure, buffers overflow and data is lost.

• Pull-Based Consumers: Consumers request data at their own pace, naturally limiting ingestion rate
• TCP Flow Control: Leverages network-level window sizing to throttle producers
• Credit-Based Systems: Consumers grant a fixed number of 'credits' to producers, replenishing them as messages are processed

In a factory with thousands of high-frequency sensors, backpressure ensures a downstream ML model doesn't crash under a burst of vibration telemetry during a machine startup.