Windowing

Fixed-size, non-overlapping time intervals that partition a stream into discrete, contiguous buckets. Each event belongs to exactly one window.

• Mechanism: A window of size T opens and closes at strict, periodic boundaries (e.g., every 5 minutes)
• Key property: Zero overlap between adjacent windows
• Use case: Computing hourly sales aggregates where each hour is independent
• Example: GROUP BY TUMBLE(event_time, INTERVAL '1' HOUR) in streaming SQL
• Contrast: Unlike sliding windows, tumbling windows never share events across boundaries

Fixed-size windows that advance forward by a period smaller than the window size, creating overlapping intervals. Also called sliding windows in some frameworks like Apache Flink.

• Mechanism: Defined by two parameters—window size and hop (advance) interval
• Key property: Events belong to multiple windows when hop < size
• Use case: Moving averages, such as

Dynamic windows that group events separated by periods of inactivity, capturing user behavior sessions without predefined boundaries.

• Mechanism: A session starts with an event and extends as long as subsequent events arrive within a gap timeout
• Key property: Window size is variable and data-driven, not fixed by time
• Use case: Grouping all clicks, page views, and actions from a single user visit
• Example: A 30-minute inactivity gap defines session boundaries in web analytics
• Implementation: Requires sessionization logic and watermark handling for late-arriving events
• Challenge: Sessions can span hours, requiring state management for long-lived windows

Windows defined by the number of events rather than temporal boundaries, triggering computation when a threshold is reached.

• Mechanism: A window closes after accumulating exactly N events
• Key property: Window duration is variable; closure depends on event rate
• Use case: Batching exactly 100 transactions for bulk processing or rate limiting
• Example: Triggering fraud analysis after every 50 payment events from a merchant
• Variation: Count-sliding windows compute aggregates over the last N events, updating with each new arrival
• Risk: Low-traffic keys may never reach the threshold, requiring timeout-based eviction policies

Watermarks are the mechanism for handling event-time skew in distributed stream processing, declaring when a window is complete despite out-of-order or late-arriving data.

• Mechanism: A watermark is a timestamp that advances through the stream, asserting that all events with timestamps less than the watermark have arrived
• Key property: Determines when window results are emitted and state can be cleaned up
• Allowed lateness: Configures how long after the watermark a window accepts late events, updating previously emitted results
• Example: A watermark lag of 10 seconds means the system waits 10 seconds for stragglers before finalizing a window
• Tradeoff: Lower watermark lag reduces output latency but increases data loss risk; higher lag improves completeness at the cost of freshness

The backend infrastructure for storing and cleaning up intermediate aggregation state as windows open, accumulate events, and expire.

• Mechanism: Each active window maintains a state store (in-memory or RocksDB-backed) holding partial aggregates
• Key property: State grows with the number of open windows and key cardinality
• Eviction: State is purged when a window's allowed lateness period expires, preventing unbounded memory growth
• Use case: Maintaining per-user session aggregates across millions of concurrent sessions
• Challenge: High-cardinality keys (e.g., user IDs) combined with long windows can cause state to balloon, requiring RocksDB spillover and careful capacity planning
• Pattern: Combine with TTL (Time-to-Live) configurations to enforce hard cleanup deadlines