Checkpointing

Checkpointing periodically serializes the entire state of a stream operator—such as aggregations, counters, or window buffers—and writes it to a durable, distributed file system like HDFS, S3, or GCS. This snapshot includes the exact offset positions in the source partitions, creating a consistent recovery point. In the event of a failure, the operator is restored from this snapshot, and processing resumes from the stored offsets, guaranteeing exactly-once semantics when combined with idempotent sinks. The frequency of this operation is a critical tuning parameter: too frequent, and I/O overhead degrades throughput; too infrequent, and recovery time increases.

Modern stream processors like Apache Flink implement checkpointing using the Chandy-Lamport algorithm adapted for continuous data flows. The job manager injects special records called checkpoint barriers into the data stream at the sources. These barriers flow through the entire directed acyclic graph (DAG) of operators, separating the stream into the portion that belongs to the current snapshot and the portion that belongs to the next. When an operator receives a barrier on all its input channels, it triggers a local state snapshot. This mechanism ensures a globally consistent snapshot across all parallel subtasks without stopping the entire pipeline.

Two primary strategies exist for managing state snapshots:

• Full Checkpointing: The entire state is written to storage on every cycle. This is simple but becomes prohibitively expensive for large state sizes, causing significant backpressure.
• Incremental Checkpointing: Only the changes to the state since the last checkpoint are persisted. This is the default in systems like Flink using RocksDB as a state backend. It dramatically reduces network I/O and storage costs, enabling checkpointing of multi-terabyte states at sub-second intervals. The trade-off is increased complexity in recovery, which requires replaying a chain of incremental files.

Standard aligned checkpointing requires barriers to wait at the input gates of an operator until all upstream channels have delivered their barrier. Under backpressure, a slow channel can stall this process indefinitely, causing the checkpoint duration to spike. Unaligned checkpointing solves this by allowing barriers to overtake buffered data. In-flight data that has not yet been processed is included as part of the channel state in the snapshot. This decouples checkpoint completion time from end-to-end latency, ensuring stable and predictable checkpoint durations even under heavy load, which is critical for latency-sensitive production pipelines.

A savepoint is a manually triggered, externally managed checkpoint that serves a fundamentally different purpose from automatic failure recovery. While checkpoints are owned by the system and cleaned up automatically, savepoints are owned by the user and are immutable. They enable critical operational workflows:

• Application Upgrades: Stop a job, upgrade the code or parallelism, and restart from a savepoint with zero data loss.
• A/B Testing: Fork a production stream into a parallel experimental pipeline from a known consistent state.
• Cluster Migration: Move a stateful job between different clusters or versions of the processing framework.

The state backend is the pluggable component that determines where and how the operator state is stored during processing and checkpointing. The choice directly impacts performance and scalability:

• HashMapStateBackend: Stores state as objects on the JVM heap. Offers the lowest latency for small to medium states but is limited by heap size and vulnerable to garbage collection pauses.
• EmbeddedRocksDBStateBackend: Stores state in an embedded RocksDB instance on local disk. It supports massive states that exceed available memory by spilling to SSD, making it the standard for production workloads with large keyed states. Checkpointing flushes the local database to durable storage.