Watermark

A watermark is fundamentally tied to event time, the timestamp when an event actually occurred in the real world. This is distinct from processing time, when the system ingests the event. Watermarks track the progress of event time, allowing the system to handle events that arrive out-of-order from the network. For example, an event with an event time of 12:05:00 might be processed at 12:07:00 due to network latency.

A watermark establishes a frontier in the event time stream. It is a monotonically increasing timestamp that asserts: "No more events with an event time less than T are expected." Events arriving with a timestamp earlier than the current watermark are considered late data. Systems can be configured with a allowed lateness period, where late data within a grace period can still be incorporated into window results, after which it may be directed to a side output for analysis.

The primary function of a watermark is to determine when a time-based window's computation can be finalized and its result emitted. For a window covering event times [12:00, 12:05), the system waits for the watermark to pass 12:05 (plus any configured allowed lateness) before concluding that all relevant data has likely arrived. This enables correct, deterministic results from out-of-order streams, as opposed to emitting speculative results based solely on processing time.

Watermarks can be generated in two primary ways:

• Perfect Watermarks: Require complete knowledge of the event stream (e.g., from a deterministic source like a replayable log). This is often impractical.
• Heuristic Watermarks: The common approach, where watermarks are inferred by observing event timestamps and tracking source progress. A simple heuristic is: Current Watermark = Max Observed Event Time - Fixed Delay. More advanced systems like Apache Flink or Google Cloud Dataflow use source-specific operators to create these estimates, which balance latency with completeness.

Configuring watermarks involves a fundamental trade-off. An aggressive (fast-advancing) watermark reduces output latency but increases the risk of incorrectly declaring data complete, leading to missed late data and inaccurate results. A conservative (slow-advancing) watermark increases result completeness and accuracy but introduces higher latency before window results are emitted. Tuning this is critical for meeting specific Service Level Objectives (SLOs) for data freshness.

Watermarks are essential for managing state in long-running streaming jobs. As the watermark advances, the system can trigger timer-based operations and perform state cleanup for windows that are now guaranteed to be complete. This prevents unbounded state growth. The combination of watermarks, checkpointing (for fault tolerance), and state management forms the backbone of robust, exactly-once stream processing architectures.

Event time is the timestamp indicating when an event or transaction actually occurred in the real-world system that generated the data. This is distinct from processing time and is the fundamental temporal domain for watermarks. Watermarks are defined relative to event time to reason about data completeness.

• Example: A sensor reading timestamped at 2024-05-15T10:00:00Z has that as its event time, even if it arrives at the processing system at 10:05.
• Critical for: Defining windows, calculating latencies, and ensuring correct temporal aggregations.

Late data refers to events that arrive at a stream processing system after the system's watermark has already advanced past the event's timestamp. Handling late data is a primary design consideration for stateful streaming applications.

• Causes: Network delays, clock skew, or mobile device offline periods.
• Handling Strategies: Systems can use allowed lateness configurations to hold window state open, route late events to a side output for special processing, or simply discard them.
• Impact: Unhandled late data can lead to incorrect results in windowed computations.

Windowed aggregation is a core stream processing operation that groups and computes aggregates (sum, count, average) over events that fall within a defined time-based window. Watermarks are the primary mechanism for determining when a window is complete and its aggregation can be emitted.

• Window Types: Tumbling windows (fixed, non-overlapping), sliding windows (overlapping), and session windows (activity-based).
• Triggering: A watermark passing the end timestamp of a window triggers the computation and output of the aggregated result for that window.

Exactly-once semantics is a processing guarantee that ensures each event in a data stream is processed effectively and its resulting state updates are applied precisely one time, despite potential failures and retries. Watermarks interact closely with the state checkpointing mechanisms that enable this guarantee.

• Mechanism: Systems like Apache Flink use distributed snapshots (checkpoints) aligned with watermark generation to create consistent global state.
• Role of Watermark: Checkpoints often wait for a watermark to pass, ensuring all relevant events for a period are included in the snapshot, enabling deterministic recovery.

Checkpointing is the process of periodically saving the state of a long-running, stateful data processing job to durable storage. This enables recovery from failures by restarting from the last saved consistent state rather than reprocessing the entire stream. Watermarks coordinate with checkpoints to ensure state consistency.

• Alignment: To ensure recovery is deterministic, a checkpoint barrier is propagated through the dataflow only after the watermark for a given time has been processed, guaranteeing all events up to that point are included.
• Trade-off: Frequent checkpoints reduce recovery time but increase runtime overhead.

Backpressure is a flow control mechanism in data streaming systems where a fast data source is signaled to slow down its data emission rate to match the processing capacity of a slower downstream consumer. This prevents system overload, queue blow-ups, and out-of-memory errors. Watermark progression can be indirectly affected by backpressure.

• Cause: A bottleneck in processing, network I/O, or sink writing speed.
• Effect on Watermarks: If a task is stalled due to backpressure, it cannot process watermarks from its inputs, causing watermark progression to stall across the entire dataflow graph. This can delay window computations and output.