Apache Flink

Flink treats batch processing as a bounded subset of stream processing, eliminating the need for separate Lambda architectures. A single application can handle both real-time event streams and historical data reprocessing using the same codebase and APIs. This unification simplifies operational complexity and ensures consistent business logic across all data processing workloads.

Flink guarantees exactly-once semantics through distributed, asynchronous checkpointing based on the Chandy-Lamport algorithm. When a failure occurs, the system rolls back all operators to the last consistent checkpoint and resumes processing without data duplication or loss. This is critical for financial transactions, billing systems, and inventory management where correctness is non-negotiable.

Flink natively supports event time semantics, processing records based on when they occurred in the real world rather than when the system received them. Watermarks track event time progress and define thresholds for handling out-of-order data. This enables accurate windowed aggregations even when events arrive late or in non-sequential order—essential for IoT telemetry, user behavior analytics, and sensor networks.

Flink provides a comprehensive set of built-in window assigners for dividing unbounded streams into finite chunks:

• Tumbling windows: Fixed-size, non-overlapping intervals
• Sliding windows: Fixed-size windows with a configurable slide interval
• Session windows: Dynamic windows based on periods of inactivity
• Global windows: Custom trigger-driven windows Each window type supports both event time and processing time semantics with custom triggers and evictors.

Flink maintains application state in a managed state backend that can be stored in-memory, on-disk via RocksDB, or in external systems. State is automatically partitioned, checkpointed, and rescaled when parallelism changes. This enables powerful stateful operations like:

• Keyed state for per-key aggregations
• Operator state for source offsets and broadcast variables
• Queryable state for low-latency external reads without a database

Savepoints are user-triggered, manually managed snapshots of the entire streaming application state. Unlike automatic checkpoints, savepoints survive application code changes and Flink version upgrades. They enable:

• A/B testing of new logic on historical state
• Schema migrations without data loss
• Reprocessing from a known point in time
• Disaster recovery with minimal downtime

An architectural pattern that persists the state of a business entity as an immutable sequence of state-changing events. Instead of storing just the current state, every mutation is captured as a discrete event.

• Flink Integration: Flink consumes these event logs as unbounded streams for real-time computation
• Enables temporal queries—reconstruct any past state by replaying events
• Forms the backbone of CQRS architectures where Flink materializes read-optimized views

A technique that divides an unbounded data stream into finite, logical chunks based on temporal boundaries. Essential for performing aggregations like counts, sums, or averages over streaming data.

• Tumbling windows: Fixed-size, non-overlapping intervals (e.g., every 5 minutes)
• Sliding windows: Overlapping intervals with a defined slide period
• Session windows: Dynamic boundaries defined by periods of inactivity
• Flink provides out-of-the-box window assigners with custom trigger and eviction logic

A mechanism that tracks event-time progress in a stream processor. Watermarks declare that all events with a timestamp earlier than the watermark have likely arrived, allowing the system to close windows and emit results.

• Handles out-of-order data by specifying a maximum allowed lateness
• Flink generates watermarks periodically or punctuated based on event characteristics
• Critical for achieving correct results in distributed, unordered event streams
• Late events beyond the watermark can be handled via side outputs

A delivery guarantee ensuring that each record is processed exactly once, even in the face of failures. Prevents both data duplication and data loss in stateful computations.

• Flink achieves this through distributed checkpointing and transactional sinks
• Uses the Asynchronous Barrier Snapshotting algorithm to create consistent global snapshots
• Requires idempotent sinks or two-phase commit protocol for end-to-end guarantees
• Essential for financial calculations, billing, and inventory management

A fault-tolerance mechanism that periodically saves the state of all operators in a Flink job to durable storage. Enables recovery from failures without data loss or complete reprocessing.

• Barrier-based: Special markers injected into the data stream trigger state snapshots
• Stored in distributed file systems like HDFS, S3, or GCS
• Incremental checkpointing reduces overhead by only saving state changes
• Enables savepoints—user-triggered checkpoints for upgrades and A/B testing

A software architecture pattern that treats all data as a stream, using a single stream processing engine for both real-time and batch processing. Eliminates the need for a separate batch layer.

• Flink's unified batch and stream processing makes it the ideal engine for Kappa architectures
• Reprocessing historical data uses the same code as real-time logic
• Simplifies operational complexity by maintaining one codebase, one framework
• Contrasts with Lambda Architecture, which requires maintaining separate hot and cold paths