Watermark

Watermarks bridge the critical gap between event time (when something happened) and processing time (when the system observed it). In distributed systems, these timelines diverge due to network latency, clock skew, and backpressure. A watermark is a monotonically increasing timestamp that declares the system's best estimate of event time progress. It allows operators to say, 'I have observed all data up to 12:01:00,' even if the processing clock reads 12:05:00. This decoupling is essential for deterministic, reproducible results regardless of when computation executes.

The primary purpose of a watermark is to define a grace period for late-arriving data. Without watermarks, a window operator would never know when to materialize results. Key behaviors include:

• Heuristic Generation: Watermarks are typically derived by observing the maximum event timestamp seen so far, minus a configurable allowed lateness bound.
• Idleness Handling: If a source partition goes silent, the watermark must not stall. Systems like Apache Flink use idle timeout mechanisms to mark partitions as idle and advance the watermark based on other active partitions.
• Late Data Routing: Events arriving after the watermark has passed their window are considered late. They can be discarded, logged to a side output, or used to issue corrective updates depending on the processing mode.

In a directed acyclic graph (DAG) of stream operators, watermarks flow downstream alongside data. When an operator consumes multiple input streams, it must compute an input watermark by taking the minimum of all upstream watermarks. This conservative approach ensures correctness: a multi-input join or union cannot safely process up to time T until all its inputs have reached T. This propagation logic is critical for maintaining exactly-once semantics in stateful operations and prevents premature window firings that would produce incomplete results.

Stream processors support two generation strategies:

• Periodic Watermarks: Generated at a fixed interval (e.g., every 200ms). The generator inspects the current maximum event time and emits a watermark. This is efficient for high-throughput streams where per-event overhead is prohibitive.
• Punctuated Watermarks: Generated in response to specific events in the stream. A sensor might emit a 'batch complete' marker, or a data file might contain an end-of-file record. This provides precise, data-driven progress tracking but requires the source to embed such signals. The choice involves a trade-off between latency (how quickly a window fires) and computational overhead.

The watermark defines the point of no return for a window, but systems often allow a configurable allowed lateness period beyond the watermark. During this grace period, a window remains in memory, and late events update the result. Once allowed lateness expires, the window state is cleaned up. Events arriving after this final deadline are routed to a side output—a separate stream that acts as a dead letter queue for late data. This pattern enables:

• Corrective updates to downstream sinks.
• Auditability of data quality issues.
• Separate business logic for processing stragglers without blocking the main pipeline.

In a distributed system, different source partitions progress at different rates. A single slow partition can hold back the entire watermark, causing watermark skew and delaying all window outputs. Mitigation strategies include:

• Idle source detection: Automatically marking stalled partitions as idle after a timeout.
• Watermark alignment: In structured streaming, applying a maximum acceptable skew between partitions.
• Speculative execution: Proactively processing data assuming a partition is complete, with rollback mechanisms if late data later arrives. Without these techniques, a single straggling Kafka partition can stall a global aggregation indefinitely.