Change Data Capture (CDC)

CDC operates at the row level, capturing individual INSERT, UPDATE, and DELETE operations. Unlike batch snapshots that compare entire tables, CDC emits only the delta—the specific rows that changed. This granularity enables precise audit trails and minimizes network payload. For example, if a single customer record in a 10-million-row table changes, CDC propagates only that one row rather than rescanning the entire dataset.

The most efficient CDC implementations read directly from the database transaction log (e.g., MySQL binlog, PostgreSQL WAL, SQL Server transaction log). This approach imposes near-zero overhead on the source database because it reads sequential log files rather than querying live tables. Log-based capture also preserves the exact commit order of transactions, ensuring downstream replicas maintain causal consistency with the source system.

CDC enables sub-second latency between a committed transaction in the source and its availability in downstream systems. This real-time characteristic supports use cases like:

• Live reporting dashboards that reflect current operational state
• Cache invalidation triggered immediately on data mutation
• Event-driven microservices that react to data changes as they occur This contrasts sharply with batch ETL, which introduces inherent delays measured in minutes or hours.

Production CDC pipelines implement at-least-once delivery with idempotent consumers to handle network failures gracefully. Each change record carries a sequence number or log position (e.g., LSN in SQL Server, SCN in Oracle), allowing consumers to deduplicate and resume from the last acknowledged position. This ensures exactly-once processing semantics even when brokers or consumers restart mid-stream.

CDC systems must accommodate schema drift as source tables evolve. When a column is added, renamed, or dropped, the CDC pipeline must propagate these changes without breaking downstream consumers. Advanced implementations integrate with a schema registry (e.g., Confluent Schema Registry) to manage versioned schemas and enforce compatibility rules, ensuring that a new nullable column doesn't crash a consumer expecting the old schema.

A single CDC stream can fan out to multiple heterogeneous sinks simultaneously. One source database change can populate:

• A search index (Elasticsearch) for full-text queries
• A data warehouse (Snowflake, BigQuery) for analytics
• A cache (Redis) for low-latency reads
• A stream processor (Kafka Streams, Flink) for real-time aggregations This decouples producers from consumers, enabling each team to consume the same truth stream independently.