Change Data Capture (CDC)

The most efficient CDC method reads directly from the source database's transaction log. This approach captures every INSERT, UPDATE, and DELETE operation with minimal overhead on the source system.

• Mechanism: Parses native database redo or write-ahead logs
• Advantage: Zero impact on application-level performance
• Examples: PostgreSQL logical decoding, MySQL binlog reader, Oracle LogMiner

An older pattern that uses database triggers to fire on data modification events. Each change invokes a procedure that writes the delta to a shadow table or queue.

• Drawback: Adds overhead to every transaction on the source
• Use Case: Legacy systems without native log access
• Trade-off: Simple to implement but introduces performance degradation

CDC systems must preserve the exact commit order of transactions as they occurred on the source. This guarantees that downstream consumers see changes in the same causal sequence.

• Total Order: Single-threaded consumption for strict consistency
• Partitioned Order: Parallel consumption per shard or table
• Critical for: Maintaining referential integrity in replicated systems

Production schemas change over time. A robust CDC pipeline must gracefully handle DDL events like adding columns or altering types without breaking downstream consumers.

• Schema Registry Integration: Avro or Protobuf schemas versioned centrally
• Backward Compatibility: New consumers read old events
• Forward Compatibility: Old consumers read new events with defaults

CDC pipelines typically guarantee at-least-once semantics, meaning every change is delivered but duplicates may occur during failures. This is a pragmatic trade-off for high throughput.

• Checkpointing: Offset tracking in source logs to resume from failure
• Idempotent Consumers: Downstream systems must deduplicate by primary key and timestamp
• Exactly-once: Achievable only with two-phase commit coordination, which adds latency

CDC systems bootstrap by taking a consistent baseline snapshot of the source, then seamlessly transition to streaming incremental changes from that point forward.

• Watermark: A marker in the log indicating the exact moment the snapshot completed
• Backfill: Historical data loaded in bulk before live tailing begins
• Consistency: Snapshot must be taken under a lock or using MVCC to avoid gaps

The end-to-end tracking of data's origin, transformations, and movement through pipelines. CDC strengthens data lineage by providing a granular, auditable record of every mutation—every insert, update, and delete—as it propagates from source to sink. This creates an unbroken chain of custody that allows quantitative engineers to trace any anomalous trading signal back to its exact source record and transformation step, satisfying both debugging and regulatory audit requirements.

A method of tracking and analyzing streams of information to derive conclusions in real time. CDC feeds act as the raw event source for CEP engines, which apply pattern-matching rules across multiple high-velocity change streams. In algorithmic trading, a CEP system might consume CDC events from market data feeds and order management systems simultaneously to detect arbitrage windows or regulatory violations within microseconds of the triggering database change.

The ability to automatically adapt a data system's structure to handle changes in the format of incoming data over time. A robust CDC implementation must gracefully handle schema evolution—such as a new column added to a source table—without breaking downstream consumers. Techniques include:

• Schema registry integration to version and resolve compatibility
• Default value injection for missing fields in old consumers
• Dead letter queues for irreconcilable schema conflicts

A change in the statistical properties of a model's input data over time, silently degrading predictive accuracy. CDC pipelines serve as an early warning system for data drift by continuously streaming fresh production data to monitoring systems. When the distribution of values captured via CDC diverges from the training baseline—such as a sudden shift in trade volumes or order book depth—alerts trigger model retraining before financial losses accumulate.

The Financial Information eXchange protocol, a standardized messaging specification for electronic communication of trade-related messages. While FIX handles external communication between institutions, CDC handles internal state propagation. A modern trading architecture often bridges the two: a CDC connector captures a new order in the internal database and publishes it, while a FIX engine translates that event into a standardized message for transmission to an exchange or counterparty.