Glossary

Change Data Capture

Change Data Capture (CDC) is a software design pattern that identifies, tracks, and delivers row-level modifications (inserts, updates, deletes) made to source data in operational databases, enabling real-time synchronization with downstream systems.

Get in touch Learn more

Data scientist building training data pipeline on laptop, data preprocessing visible, technical workspace.

REAL-TIME DATA SYNCHRONIZATION

What is Change Data Capture?

Change Data Capture (CDC) is a software design pattern that identifies, tracks, and delivers row-level data modifications—inserts, updates, and deletes—from a source database to downstream systems in real time.

Change Data Capture is a method for ingesting database mutations by reading the transaction log, avoiding the latency and load of batch queries. It captures every INSERT, UPDATE, and DELETE as an event, enabling immediate synchronization of the online store with operational systems.

In feature engineering, CDC ensures feature freshness by streaming changes directly to the materialization pipeline. This guarantees that point-in-time correctness is maintained and that models serve vectors reflecting the current state of the entity, not a stale snapshot.

DATA INGESTION PATTERNS

Key Characteristics of CDC

Change Data Capture provides a low-latency mechanism for detecting and propagating row-level mutations from operational databases into analytical and serving systems.

Log-Based Capture

The most efficient CDC method reads directly from the database transaction log. This approach captures INSERT, UPDATE, and DELETE operations with minimal impact on source system performance. Unlike trigger-based methods, log-based CDC does not require schema changes to operational tables.

PostgreSQL: Uses logical replication slots and the WAL (Write-Ahead Log)
MySQL: Consumes the binlog via connectors like Debezium
SQL Server: Leverages the native transaction log via LSN (Log Sequence Number)

< 1 ms

Overhead on Source

Sub-second

Propagation Latency

Schema Evolution Handling

CDC pipelines must gracefully manage schema drift in the source database without breaking downstream consumers. When a column is added or a data type is widened, the CDC connector emits a schema change event to update the registry.

Avro and Protobuf serialization with a schema registry ensure backward compatibility
Dead Letter Queues route malformed records for inspection without halting the stream
Feature Store Integration: Updated schemas trigger automatic re-materialization of affected feature groups

Exactly-Once Semantics

Ensuring that every row mutation is processed exactly once is critical for maintaining accurate feature values in the online store. CDC pipelines coordinate offsets between the source log and the downstream sink.

Idempotent Writes: Feature stores use upsert operations keyed by entity ID and timestamp
Two-Phase Commit: Kafka Connect sinks flush offsets only after the target system acknowledges the write
Deduplication: Ingested events carry a unique transaction ID to filter out replayed records

Filtering and Transformation

Not every database change is relevant for feature engineering. Single Message Transforms (SMTs) and stream processors filter and reshape events before they reach the feature store.

Column Filtering: Drop sensitive PII columns or irrelevant metadata before ingestion
Routing: Dispatch events to different topics based on table name or operation type
Enrichment: Join CDC events with static reference data to hydrate foreign keys into human-readable values

Snapshot vs. Streaming Mode

CDC connectors operate in two distinct phases. An initial snapshot captures the full state of a table, followed by continuous streaming of incremental changes.

Snapshot: Acquires a consistent read lock to export all existing rows with their current values
Streaming: Transitions seamlessly to tailing the transaction log after the snapshot completes
Resumption: If the connector fails, it resumes from the last committed log offset without re-snapshotting

Debezium and Kafka Connect

The de facto open-source CDC ecosystem relies on Debezium connectors deployed on Kafka Connect clusters. This architecture decouples source database concerns from downstream consumption.

Connectors: Debezium provides dedicated plugins for MySQL, PostgreSQL, MongoDB, and SQL Server
Kafka Topics: Each captured table maps to a dedicated topic for ordered, replayable event streams
Feature Store Sink: A custom Kafka Connect sink writes transformed events directly into the online store's low-latency database

EXPLORE

CHANGE DATA CAPTURE

Frequently Asked Questions

Explore the core concepts of Change Data Capture (CDC), a critical pattern for streaming row-level mutations from operational databases into real-time feature stores to power low-latency online inference.

Change Data Capture (CDC) is a data integration pattern that identifies, captures, and delivers row-level changes—INSERT, UPDATE, and DELETE operations—made to source database tables in real-time. Instead of relying on bulk batch extracts that snapshot entire tables, CDC monitors the database's transaction log (such as MySQL binlog, PostgreSQL WAL, or SQL Server transaction log) to detect mutations as they are committed. When a row changes, the CDC system immediately serializes the event into a structured message, typically JSON or Avro, and publishes it to a distributed streaming platform like Apache Kafka. Downstream consumers, such as a feature store's online serving layer, ingest these events to update specific feature values—like a user's last-clicked product or a real-time inventory count—with minimal latency, often measured in milliseconds. This log-based approach is non-intrusive to the source application, ensuring that operational database performance remains unaffected while providing a continuous, ordered stream of truth for machine learning pipelines.

Enabling Efficiency, Speed & Accuracy

Intelligent Analysis, Decision & Execution

We build AI systems for teams that need search across company data, workflow automation across tools, or AI features inside products and internal software.

Talk to Us

Search across company data

Give teams answers from docs, tickets, runbooks, and product data with sources and permissions.

Useful when people spend too long searching or get different answers from different systems.

Enterprise searchRAGPermissions

Automate internal workflows

Use AI to route work, draft outputs, trigger actions, and keep approvals and logs in place.

Useful when repetitive work moves across multiple tools and teams.

AI agentsWorkflow automationGovernance

Add AI to products and internal tools

Build assistants, guided actions, or decision support into the software your team or customers already use.

Useful when AI needs to be part of the product, not a separate tool.

AI integrationDecision supportModel routing

ECOSYSTEM

Related Terms

Change Data Capture is a critical data engineering pattern that feeds real-time feature stores. Explore the interconnected components that rely on CDC to power low-latency machine learning inference.

Streaming Features

Feature values computed incrementally on real-time event data with low latency. CDC provides the raw, row-level change events that stream processing engines like Apache Flink or Kafka Streams consume to compute these features.

Captures immediate user intent signals
Enables sub-second feature freshness
Powers real-time personalization models

Online Store

The low-latency database component of a feature store designed to serve pre-computed feature vectors during real-time prediction. CDC pipelines are the primary mechanism for keeping the online store synchronized with the operational database.

Typically backed by Redis or DynamoDB
Must reflect source-of-truth changes in milliseconds
CDC minimizes the write-path latency from OLTP to online store

Materialization

The process of pre-computing feature values from source data and persisting them into an online or offline store. CDC triggers incremental materialization, avoiding costly full-table recomputation.

Incremental materialization reduces compute waste
Ensures feature vectors are fresh for inference
Tightly coupled with feature freshness SLAs

Feature Freshness

A metric defining the maximum acceptable age of a feature value in the online store. CDC directly impacts this SLA by minimizing the propagation delay from an operational write to the feature store update.

Measured as the lag between source commit and online store availability
Tight freshness (< 1 sec) requires efficient CDC and streaming
Stale features cause models to act on outdated context

Point-in-Time Correctness

A data engineering guarantee that feature values used for model training are reconstructed exactly as they existed at a specific historical timestamp. CDC logs provide the immutable, ordered changelog required to build these time-travel consistent training datasets.

Prevents data leakage from future information
Relies on the transaction log as the source of truth
Essential for accurate offline evaluation

Feature Lineage

The tracked metadata that maps the complete lifecycle of a feature from its raw source data through transformations to its consumption by a model. CDC captures the initial hop from the operational database, forming the root of the lineage graph.

Tracks origin down to the specific database row and LSN
Enables precise auditing and debugging
Critical for governance in regulated industries

About the author

Prasad Kumkar

CEO & MD, Inference Systems

Prasad Kumkar is the CEO & MD of Inference Systems and writes about AI systems architecture, LLM infrastructure, model serving, evaluation, and production deployment. Over 5+ years, he has worked across computer vision models, L5 autonomous vehicle systems, and LLM research, with a focus on taking complex AI ideas into real-world engineering systems.

His work and writing cover AI systems, large language models, AI agents, multimodal systems, autonomous systems, inference optimization, RAG, evaluation, and production AI engineering.

Limited slotsGet a Free AI Consultation

How We Work

Custom AI workflows for your Business

One-fit-all AI don't work for modern businesses. At Inferensys, we aim to understand your business & custom requirements; which we use to define most efficient agentic workflows, the data, and the tools for your business.