Watermark

Watermarks are fundamentally tied to event time, the timestamp when an event actually occurred in the real world, as opposed to processing time, when the system observes the event. Since events can arrive late or out-of-order, the system cannot wait indefinitely. A watermark is a heuristic that estimates completeness, allowing the system to reason about event time progress and trigger computations like window aggregation. For example, a watermark of 12:05 means the system believes all events with timestamps up to 12:05 have been ingested.

A watermark is a heuristic estimate, not a guarantee. It is calculated based on observed data patterns, source timestamps, and network delays. Systems typically generate watermarks by observing the maximum event timestamp seen, minus a configurable allowable lateness delay (e.g., 10 seconds). This creates a trade-off: a smaller delay reduces output latency but risks discarding more genuinely late data, while a larger delay increases latency but improves result completeness. Late data arriving after the watermark has passed its window may be handled via side outputs or dropped, depending on the system's configuration.

The primary function of a watermark is to trigger the evaluation of event-time windows. A window (e.g., a 5-minute tumbling window) is considered ready for computation when the watermark advances past the window's end time. For a window covering 12:00-12:05, the system will close the window and emit a result once the watermark passes 12:05. This mechanism allows for deterministic, repeatable results based on when events happened, not when they were processed, which is critical for accurate time-series analytics over unbounded data streams.

Watermarks can be generated at different points in the pipeline:

• Source-Generated: The most accurate method. The data source (e.g., Apache Kafka, Google Pub/Sub) attaches watermarks based on its knowledge of partition progress. This is common in log-based systems.
• Operator-Generated: The stream processing engine (like Apache Flink or Apache Beam) generates watermarks internally by observing timestamps from all upstream sources and propagating the minimum watermark across parallel streams. This ensures downstream operators have a consistent view of time progress.
• Punctuated vs. Periodic: Watermarks can be emitted in response to specific events (punctuated) or at regular time intervals (periodic).

A critical challenge occurs when a data source or partition becomes idle (stops sending data). If one partition in a parallel stream is idle, its watermark stops advancing. Since the system-wide watermark is often the minimum across all parallel inputs, a single idle source can stall the entire pipeline's time progress, preventing window triggers. Modern systems like Apache Flink have idleness detection to temporarily exclude idle sources from the watermark calculation, allowing time to advance based on active sources. This prevents pipeline stalls due to sporadic data sources.

Watermarks are a key component in achieving exactly-once processing semantics in stateful stream processors. For a system to provide consistent, fault-tolerant results, it must know when to take a consistent snapshot (checkpoint) of its state. Watermarks signal a point in the event-time stream where the system can consider a set of windows 'complete enough' to checkpoint their intermediate state. This coordination between watermarks, checkpointing, and state backend storage (like RocksDB) enables the system to recover from failures and produce deterministic, non-duplicated results.

A fundamental distinction in stream processing. Event time is when the event actually occurred in the real world (embedded in the data payload). Processing time is when the event is observed by the streaming system. Watermarks track progress in event time, which is essential for accurate, out-of-order analysis.

• Example: A sensor emits a reading at 10:00:00 (event time) but network delay causes it to arrive at the stream processor at 10:00:05 (processing time).
• Why it matters: Windows and aggregations based on event time produce correct, reproducible results, even when data arrives late.

A core stream processing operation where data is grouped into finite chunks (windows) for aggregation. Watermarks signal when a window can be considered complete and its results emitted.

• Types: Tumbling (fixed, non-overlapping), Sliding (overlapping), and Session (activity-based) windows.
• Role of Watermark: A watermark with timestamp T indicates the system believes no more data with event time < T will arrive. This allows the system to close and trigger computation for windows whose end time is < T.
• Example: A 1-minute tumbling window for 10:00-10:01 closes when the watermark passes 10:01.

Data that arrives after the watermark has passed its event timestamp. Real-world systems must handle this to maintain correctness.

• Allowed Lateness: A configurable grace period after a window closes. Data arriving within this period triggers a late firing, updating the window's result.
• Side Outputs: A mechanism to route data that arrives after the allowed lateness period for separate handling (e.g., to a dead-letter queue for analysis).
• Trade-off: Larger allowed lateness increases result accuracy but delays output and increases state size.

A fault-tolerance mechanism where a streaming system periodically saves its state (including current watermark timestamps) to durable storage. This enables recovery from failures with exactly-once or at-least-once processing guarantees.

• Relationship to Watermarks: During recovery, the system restores not only operator state but also the watermark progress, ensuring temporal consistency is maintained.
• Barriers: In frameworks like Apache Flink, checkpoints are implemented using barriers that flow through the data stream, aligning snapshots across parallel operators.