Watermark

A watermark is a monotonically increasing timestamp that flows through the stream alongside data records. It declares that all events with a timestamp less than the watermark value have been observed, allowing operators to close windows and emit results. Unlike wall-clock time, watermarks track event time, which is the time embedded within the data itself. This decoupling is essential for handling out-of-order events caused by network delays, clock skew, or distributed system latencies. The watermark advances based on observed event timestamps, often using a heuristic such as the maximum observed timestamp minus a configured allowed lateness bound.

Watermarks are generated using periodic or punctuated strategies:

• Periodic Watermarks: A timer fires at fixed intervals, inspecting the current maximum event timestamp and emitting a new watermark. This is the most common approach in systems like Apache Flink.
• Punctuated Watermarks: Specific events in the stream trigger watermark generation. For example, a special 'end-of-session' marker or a sensor reading indicating a batch is complete.
• Idle Source Handling: If a source partition becomes idle (no data), it must be marked as idle to prevent it from holding back the watermark for the entire pipeline. Without this, a single idle partition can stall all window outputs indefinitely.

In a distributed stream processor with multiple parallel source tasks, the operator watermark is the minimum of all input watermarks. This ensures correctness: a downstream operator cannot close a window until every upstream shard has confirmed it has processed all relevant data. This watermark alignment is a critical bottleneck. A single slow or straggling source task can stall the entire pipeline's watermark progress. Advanced systems implement watermark skew mitigation techniques, such as ignoring idle sources or applying per-partition watermarks with a configurable alignment timeout.

While often conflated, watermarks and triggers serve distinct roles. A watermark is a declaration of input completeness. A trigger is a mechanism that determines when window results are materialized. Watermarks are the default and most common trigger, but triggers can also be based on:

• Processing time: Fire every N seconds regardless of data completeness.
• Element count: Fire after a certain number of records.
• Composite triggers: Fire on watermark or early, then update on late arrivals. This separation allows systems to provide speculative early results before the watermark arrives, improving perceived latency while still providing a definitive final result.

A continuous computation technique that calculates metrics over a dynamically updating, overlapping time interval. Unlike fixed windows, sliding windows require watermarks to determine when an event falls out of the window's scope.

• Trigger: Recalculation occurs on every new event or at periodic intervals
• Use case: Computing rolling 5-minute velocity checks for a card number
• Watermark role: Ensures late-arriving transactions are not excluded from the window if they fall within the allowed lateness threshold

A method of tracking and analyzing streams of information to identify meaningful patterns and causal relationships across multiple data sources. CEP engines rely on watermarks to define temporal boundaries for pattern matching.

• Detects composite events like '3 failed logins followed by a password reset within 60 seconds'
• Uses event time rather than processing time for accuracy
• Watermarks signal when a pattern sequence can be finalized and emitted downstream

A flow control mechanism that prevents a fast data producer from overwhelming a slower consumer. When a downstream operator is waiting for a watermark to advance before computing a window result, backpressure signals propagate upstream.

• Prevents out-of-memory errors in stateful stream processors
• Buffering strategy: Temporarily holds events until the watermark advances
• Critical in fraud pipelines where a single slow enrichment service can stall the entire scoring window

A data processing guarantee ensuring each record is processed precisely one time, eliminating data loss or duplication. Watermarks are integral to checkpointing mechanisms that enable exactly-once delivery in stateful operations.

• Checkpoint barrier: A special marker injected into the stream that aligns with watermarks
• Ensures window results are committed atomically
• Prevents duplicate fraud alerts if a streaming node fails and restarts mid-window

A performance metric indicating the maximum response time experienced by the fastest 99% of requests. In streaming fraud systems, watermark propagation delay directly impacts P99 latency for windowed aggregations.

• Tail latency: The 1% of requests slower than P99 are often caused by waiting for late data
• Watermark heuristics balance completeness against latency
• A strict watermark of 500ms ensures P99 scoring latency stays within the authorization window

A secondary queue where messages that cannot be processed successfully are routed for manual inspection. Events arriving after the watermark has advanced beyond their event time are often sent to a DLQ rather than silently dropped.

• Late data policy: Defines whether to discard, reroute, or partially update results
• Preserves audit trails for transactions excluded from real-time windows
• Enables offline reconciliation to identify systemic clock skew issues across payment terminals