Watermarking

Watermarking operates on event time, the timestamp when an event actually occurred, not processing time, when the system processes it. This distinction is critical for accurate analytics on real-world data where events arrive out-of-order. Watermarks track progress in event time, allowing the system to understand when it can safely assume all data for a period (e.g., 1:00 PM to 1:05 PM) has been seen, even if some late events are still trickling in during processing time.

A primary function of watermarking is to manage late-arriving data. The watermark is a monotonically increasing timestamp that signifies "no events with a timestamp less than X are expected." Data arriving with a timestamp older than the current watermark is considered late. Systems can define a allowed lateness period, where late data can still be incorporated into window results, after which it is typically discarded or sent to a side output for special handling.

Watermarks directly control when time-based windows (e.g., tumbling, sliding, session) are evaluated and their results emitted. A window's computation is triggered only when the watermark advances past the window's end time plus any configured allowed lateness. This provides deterministic output despite the non-deterministic arrival of streaming data. For example, a 5-minute window ending at 1:05 PM will fire when the watermark passes 1:05 PM (or later, if lateness is allowed).

Watermarks can be generated in two primary ways:

• Heuristic Watermarks: Estimate event time progress based on observed data patterns (e.g., using the maximum timestamp seen minus a fixed delay). This is common but can be incorrect, potentially causing late data to be missed or windows to fire prematurely.
• Perfect Watermarks: Require knowledge of all event timestamps in advance, which is only possible with data sources that provide explicit progress guarantees (e.g., bounded logs). These guarantee no late data.

Watermarks originate from data sources (e.g., Apache Kafka, Apache Pulsar) which can emit watermarks based on partition progress. Downstream operators (like window functions) receive these watermarks and propagate a combined watermark, which is the minimum of all input watermarks. This ensures a global view of progress across parallel streams. If one source partition stalls, the global watermark stalls, preventing windows from closing until all parallel inputs have advanced.

Watermarks enable efficient state cleanup in stateful stream processors. Once the watermark has passed the end of a window and its allowed lateness period, the system can safely garbage collect the state associated with that window. This prevents unbounded state growth. The state backend (e.g., RocksDB) relies on watermark signals to know which keyed state is no longer needed for future calculations.

A fundamental distinction in stream processing. Event Time is when an event actually occurred, embedded in its payload. Processing Time is when the system ingests and processes the event. Watermarks track progress in event time, enabling correct computations despite out-of-order arrivals and variable processing delays.

• Example: A sensor emits a reading at 10:00:00 (event time) that arrives at the pipeline at 10:00:05 (processing time) due to network latency.
• Impact: Aggregations based on processing time are inaccurate for time-based business logic; event time with watermarks is required.

The stream processing operation that watermarking enables. It groups events into finite chunks (windows) for computation (e.g., sum, average). Watermarks signal when the system can assume all data for a given event-time window has arrived, triggering the window's computation and emitting its result.

• Tumbling Windows: Fixed-size, non-overlapping windows (e.g., every 5 minutes).
• Sliding Windows: Fixed-size windows that slide by a specified period, creating overlaps.
• Session Windows: Dynamically sized based on periods of activity separated by gaps of inactivity.

A configurable tolerance period that extends beyond the watermark. It allows a window to accept and incorporate late-arriving data that falls within the allowed lateness bound, updating the previously emitted result. This is crucial for handling data that arrives significantly late without indefinitely delaying computation or bloating state.

• Mechanism: When a late element arrives within the allowed lateness, the window is re-triggered, and a corrected output (or a retraction) may be emitted.
• Trade-off: Increases state size and complexity, as windows must remain open longer.

The mechanism that determines when to emit the results of a window. While watermarks are the primary trigger for completeness, systems support multiple, flexible triggers.

• Watermark Progress: Fires when the watermark passes the end of the window (default).
• Processing-Time Timers: Fires at a specific wall-clock time, useful for providing early/approximate results.
• Punctuations / Data-Driven: Fires after observing a certain number of elements or a specific data condition.
• Composite Triggers: Combine multiple triggers (e.g., early firing every 1 second, then final firing on watermark).

The storage subsystem that manages the persistent state required for watermarks and windowing. It holds buffered events for open windows, partial aggregates, and watermark timers until they are triggered.

• In-Memory/Heap: Fast but volatile; state lost on failure.
• RocksDB: The default for many systems (e.g., Apache Flink). Offers large, disk-backed state that survives failures via checkpointing.
• External Databases: For queryable state or very large state, systems can integrate with external K/V stores.
• Performance Impact: The choice of backend directly affects throughput, latency, and recovery time.

A processing guarantee that each record in a stream affects the final output exactly once, even in the face of machine or process failures. Watermarking and windowing must be integrated with fault-tolerance mechanisms to maintain this guarantee.

• Dependency on Checkpointing: Periodic, consistent snapshots of pipeline state (including watermark progress and window buffers) are saved to durable storage.
• Recovery: On failure, the pipeline resets its source position, restores state from the last checkpoint, and recomputes. Watermarks are restored, ensuring no duplicate or missing window emissions.
• Idempotent Sinks: Often used in conjunction to handle duplicate emissions that can occur during recovery.