Event Stream Processing (ESP)

Unlike batch processing, which operates on finite datasets, ESP is designed to process unbounded data streams that have no defined end. The system continuously ingests and processes events as they arrive, maintaining application state indefinitely. This is achieved through techniques like windowing and watermarks, which provide temporal boundaries for aggregations over an infinite stream without requiring the stream itself to stop.

ESP systems maintain durable, fault-tolerant state local to the processing engine. This allows applications to remember information across individual events, enabling complex operations like:

• Joins: Correlating events from multiple streams (e.g., matching a click to an impression).
• Aggregations: Computing running counts, sums, or averages over time windows.
• Pattern Detection: Tracking sequences of events to identify a state machine transition. State is typically backed by a local embedded key-value store like RocksDB and persisted to a checkpoint for recovery.

A critical guarantee for transactional use cases, exactly-once semantics ensures that even in the event of a failure, each record's effect on the application state is applied precisely one time. This is implemented through a combination of distributed checkpointing and idempotent writes to sinks. During recovery, the system rewinds its state to the last consistent checkpoint and replays the source stream, preventing both data duplication and data loss.

ESP distinguishes between two temporal domains:

• Event Time: The timestamp when an event actually occurred at its source.
• Processing Time: The wall-clock time when the event is observed by the processing engine. Using event time is essential for correctness in distributed systems where events can arrive out of order due to network delays. Watermarks are the mechanism used to track event time progress and signal when all events up to a certain timestamp are expected to have arrived, triggering windowed computations.

ESP engines are optimized for sub-second end-to-end latency while sustaining millions of events per second. This performance is achieved through:

• Pipelined execution: Operators process data without unnecessary disk I/O.
• Asynchronous checkpointing: State snapshots are taken continuously without pausing the data stream.
• Network-efficient shuffle: Data is redistributed between parallel operator instances using optimized network stacks. This architecture enables real-time use cases like fraud detection and dynamic pricing that cannot tolerate batch-processing delays.

ESP is a foundational technology for implementing Command Query Responsibility Segregation (CQRS) and Event Sourcing patterns. In this model, state-changing operations (commands) are captured as an immutable, append-only log of events. The ESP engine consumes this log to build and maintain materialized views or projections that serve query requests. This decouples the write model from the read model, allowing for independently scalable, highly optimized query performance for real-time customer segmentation.

A core ESP operation that computes summary statistics over finite, time-bounded subsets of an infinite event stream. Rather than aggregating all historical data, windows define logical boundaries—such as tumbling, sliding, or session windows—to calculate metrics like moving averages, counts, or sums over the last 5 minutes or 100 events. This enables real-time dashboards and anomaly detection without unbounded state growth.

A mechanism that tracks event time progress in a stream and defines a threshold for how long the system waits for late-arriving data before finalizing window results. When a watermark passes the end of a window, the computation triggers output. Watermarks handle the inherent skew between event time and processing time, ensuring correctness even when events arrive out of order or delayed due to network latency.

A delivery and processing guarantee ensuring that, even in the presence of failures, retries, or network partitions, each event record is processed precisely one time. The resulting application state is computed as if no failure occurred. This is achieved through idempotent producers, transactional sinks, and distributed checkpointing—critical for financial ledgers, billing systems, and any use case where duplicate or missing records are unacceptable.

A higher-order pattern-matching paradigm that analyzes streams to infer meaningful composite events from sequences of atomic occurrences. CEP engines detect temporal patterns like 'a login failure followed by a password reset within 60 seconds' or 'a cart abandonment after 3 product views.' Unlike simple filtering, CEP correlates events across multiple streams and time windows to identify threats, opportunities, or workflow triggers.

A design pattern that identifies and tracks row-level mutations in source database tables—inserts, updates, deletes—and streams those changes in real-time to downstream systems. CDC transforms a static relational database into a live event source, enabling ESP engines to react to transactional data without batch ETL latency. Common implementations use database transaction logs to minimize overhead on the source system.

An architectural pattern where the authoritative state of a business entity is derived from an immutable, append-only sequence of all state-changing events. Instead of storing the current balance, you store every deposit and withdrawal. This provides a complete audit trail, temporal querying, and replayability—making it a natural fit for ESP systems that need to rebuild state or backtest logic against historical event streams.