Change Data Capture (CDC)

CDC captures only inserts, updates, and deletes as they occur, rather than performing full-table scans or bulk transfers. This is achieved by reading database transaction logs (e.g., MySQL's binlog, PostgreSQL's Write-Ahead Log).

• Efficiency: Drastically reduces network load and processing overhead compared to batch-based ETL.
• Low Latency: Changes are streamed with sub-second latency, enabling near real-time analytics and synchronization.
• Example: A user profile update in a source CRM database is captured as a single update event, not by re-syncing the entire user table.

The most robust CDC implementations use the database's native transaction log as the source of truth. This provides a reliable, ordered record of all committed changes.

• Non-Intrusive: Does not require triggers or schema modifications on the source database, minimizing performance impact.
• Fault Tolerance: Logs provide durability; CDC connectors can resume from the last committed offset after a failure.
• Atomicity & Order: Maintains the atomicity of transactions and the exact sequence of operations, which is critical for financial systems.

CDC transforms database changes into a stream of event messages, typically published to a message broker like Apache Kafka or Amazon Kinesis.

• Decoupled Architecture: Downstream consumers (data warehouses, search indexes, caches) subscribe to the event stream independently.
• Event Sourcing Foundation: The change stream serves as an immutable ledger of state changes, enabling event-sourced systems.
• Use Case: An e-commerce order status change event triggers updates in a Redis cache, an Elasticsearch index, and an analytics dashboard simultaneously.

CDC events are enriched with schema metadata, describing the structure of the changed data. This is often managed by a Schema Registry.

• Data Contract Enforcement: Ensures compatibility between data producers (the source DB) and consumers.
• Schema Evolution: Supports safe, backward-compatible changes to table structures (e.g., adding a nullable column) without breaking downstream pipelines.
• Format: Events often use serialization formats like Apache Avro or Protocol Buffers (Protobuf) that embed schema IDs for efficient parsing.

CDC connectors (e.g., Debezium, Kafka Connect) are stateful services that track their read position in the source log and manage connector offsets.

• Exactly-Once Processing: Advanced connectors can coordinate with the message broker to provide exactly-once semantics, preventing duplicate or lost events.
• Checkpointing: Regularly persists its position, allowing for graceful restarts and recovery from failures.
• Scalability: Connectors can be distributed and scaled horizontally to handle high-volume change streams from multiple database shards.

A complete CDC pipeline must handle the initial snapshot—the first full copy of existing data—before transitioning to incremental change streaming.

• Bootstrapping: The connector can perform a consistent, non-blocking snapshot of all tables or selected tables.
• Seamless Transition: After the snapshot, the connector automatically begins streaming incremental changes from the log position captured at the snapshot's start.
• Catch-Up Mode: If a consumer falls behind, it can reprocess historical events from the retained log in the message broker without querying the source database.

Event-Driven Architecture (EDA) is a software design paradigm where the flow of the application is determined by the production, detection, and consumption of events. CDC is a foundational enabler of EDA.

• Relationship: CDC transforms state changes in a system of record (the database) into a stream of immutable events. This stream becomes the single source of truth for propagating state across microservices, data lakes, and search indexes.
• Benefit: Enables loose coupling and real-time reactivity. Downstream services subscribe to change events without directly querying the source database, reducing load and latency.

A Data Contract is a formal agreement between data producers and consumers that specifies the schema, semantics, quality, and service-level expectations for a data product. In CDC, the change event stream is a critical data product.

• Application to CDC: Defines the exact format of change events (schema), the meaning of fields (semantics), guarantees like ordering or delivery semantics (at-least-once, exactly-once), and latency SLOs.
• Governance: Managed through tools like a Schema Registry, which enforces compatibility (e.g., backward/forward compatibility) as the source database schema evolves, preventing breaking changes for consumers.

Exactly-Once Semantics is a processing guarantee that each change event captured from the source database will be processed precisely one time by downstream systems, with no data loss and no duplication, despite potential failures in the pipeline.

• Challenge in CDC: Network failures, consumer restarts, and reprocessing can lead to duplicate events.
• Implementation: Achieved through idempotent writes at the sink and transactional coordination between reading the database log and publishing to Kafka (e.g., using Kafka transactions). This is crucial for maintaining financial or inventory accuracy in target systems.

Data Lineage is the tracking of data's origin, movement, characteristics, and transformations across its lifecycle. For CDC pipelines, lineage provides critical visibility.

• CDC Context: Tracks a record from its origin (a specific row in a source database table) through its capture as a change event, its journey across Kafka topics, and its final state in a data warehouse, cache, or search index.
• Value: Essential for auditing, debugging (e.g., tracing why a value appeared in a dashboard), impact analysis (understanding what breaks if a source schema changes), and meeting regulatory compliance requirements.