Event Time

Event time is embedded in the event payload at the point of origin and cannot be altered by downstream processing. Unlike processing time, it is impervious to network lag, backpressure, or system clock skew. This immutability makes it the only reliable basis for:

• Replaying historical data with exact fidelity
• Auditing user behavior for compliance
• Correlating events across geographically distributed producers

In distributed systems, events rarely arrive in the exact sequence they occurred. Watermarks are the primary mechanism for managing this disorder. A watermark is a threshold timestamp that advances as the system observes event time progress, declaring that all data earlier than T should have arrived. Key behaviors:

• Perfect Watermarks: Require total knowledge of input (rare in practice)
• Heuristic Watermarks: Estimate progress using bounded delay assumptions
• Late events arriving after the watermark can be discarded, logged to a side output, or used to recompute results

Event time enables meaningful temporal grouping through windowing. Without it, aggregations would reflect only the order of ingestion. Common window types include:

• Tumbling Windows: Fixed-size, non-overlapping intervals (e.g., every 5 minutes)
• Sliding Windows: Fixed-size windows that advance by a step smaller than the window size, creating overlap
• Session Windows: Dynamic windows defined by a gap of inactivity, ideal for grouping user interactions into a single session All window boundaries are calculated against the event time, not the system clock.

Because event time is a fixed attribute of the data, reprocessing a stream from a persistent log like Apache Kafka or Amazon Kinesis yields identical results every time. This property is critical for:

• Backtesting new personalization models against historical user behavior
• Correcting logic errors in a downstream processor without data loss
• Migrating stream processing jobs between frameworks (e.g., from Apache Flink to a new engine) The system must read the original event timestamps, not the time of the replay operation.

Sessionization relies entirely on event time to group discrete clicks, views, and actions into a single user session. A session window is defined by a timeout gap (e.g., 30 minutes of inactivity). The algorithm:

1. Assigns each event to a key (user ID)
2. Sorts events by event time
3. Merges events into a session if the gap between consecutive events is less than the timeout Using processing time would fracture sessions due to arbitrary delivery delays, corrupting metrics like session duration and conversion paths.

Joining multiple streams requires aligning them by event time to ensure correctness. For example, correlating an ad impression with a purchase requires matching the click event time, not the server receipt time. Interval joins in stream processors use event time bounds to match records across streams within a defined window. Without event time alignment, a fast stream would be joined with stale data from a lagging stream, producing nonsensical attribution.

The mechanism that slices an unbounded stream into finite chunks for aggregation. Without windowing, operations like SUM or AVG would never complete on an infinite stream.

• Tumbling windows: Fixed-size, non-overlapping (e.g., every 5 minutes)
• Sliding windows: Fixed-size with overlap (e.g., 10-min window every 1 min)
• Session windows: Dynamic boundaries based on inactivity gaps
• Windows are always defined against event time, not processing time

The process of grouping discrete user events into continuous sessions based on a configurable inactivity gap. A session ends when no event arrives for the defined timeout period.

• Critical for user behavior modeling in e-commerce and analytics
• Uses session windows in stream processors like Apache Flink
• Example: A user browsing products, adding to cart, and checking out within 30 minutes = one session
• Requires event time to correctly reconstruct the user's actual journey

The ability to correctly handle events that arrive late or in a different sequence than they occurred. Network latency, device clock skew, and distributed systems all cause disorder.

• Watermarks define the acceptable lateness boundary
• Events arriving after the watermark are late events and may be:
  • Discarded
  • Sent to a side output for separate handling
  • Used to update previously emitted results (with retraction support)
• Essential for exactly-once semantics in financial and retail pipelines

A fault-tolerance mechanism that periodically snapshots the entire state of a streaming operator to durable storage. On failure, the system restores from the last successful checkpoint.

• Enables exactly-once processing guarantees
• Checkpoints capture event time progress (watermarks) and operator state
• Barrier-based checkpointing in Apache Flink injects markers into the stream to create consistent snapshots without pausing processing
• Critical for stateful operations like windowing and joins