Stream Processor

Unlike batch processors that operate on finite, static datasets, a stream processor ingests a potentially infinite, continuously flowing dataset. It never assumes the data has ended. This requires an event-driven execution model where computation is triggered by the arrival of each new record, rather than a scheduled job. The processor must maintain a perpetual query that remains active indefinitely, processing data incrementally as it arrives.

Stream processors maintain local, fault-tolerant state to perform operations that require context beyond a single event. This state is stored in a state store, often an embedded RocksDB instance. Key operations requiring state include:

• Aggregations: Computing running counts, sums, or averages over windows.
• Joins: Correlating events from multiple streams based on a common key.
• Pattern Detection: Tracking sequences of events over time (e.g., abandoned cart). State is periodically persisted via checkpointing to durable storage for recovery.

A critical distinction in stream processing is the difference between event time (when the event actually occurred in the real world) and processing time (when the system observed the event). Stream processors use watermarks—special timestamps injected into the data flow—to track event-time progress. A watermark with timestamp T asserts that the system believes all events with an event time earlier than T have been observed, allowing windows to be closed and results emitted correctly even with out-of-order data.

In distributed systems, failures can cause records to be reprocessed, leading to duplicate results. Stream processors implement exactly-once state semantics to ensure that state updates and downstream outputs are applied only once, even during retries. This is achieved through atomic checkpointing combined with idempotent sinks. The processor coordinates a distributed snapshot of the entire processing graph, ensuring all operators recover to a consistent state with no gaps or duplicates in the computation.

Since streams are unbounded, aggregations must be scoped to finite time intervals called windows. Common window types include:

• Tumbling Windows: Fixed-size, non-overlapping, contiguous intervals (e.g., every 5 minutes).
• Sliding Windows: Fixed-size windows that slide by a specified interval, creating overlap (e.g., 10-minute window every 1 minute).
• Session Windows: Dynamically sized windows that group events separated by a period of inactivity, defined by a session gap. Windows are closed and results emitted when the watermark passes the window's end.

A fast upstream producer can overwhelm a slow downstream consumer. Stream processors implement backpressure as a natural flow-control mechanism. Unlike traditional message queues that buffer data, stream processors use blocking queues between operators. If a downstream operator is slow, the upstream operator's output buffer fills, causing it to block and stop reading from its own input, propagating the backpressure all the way to the source connector, effectively pausing ingestion until the bottleneck resolves.

A mechanism that slices an unbounded data stream into finite, temporal buckets for aggregation. Without windows, operations like SUM or AVG would never complete on an infinite stream.

• Tumbling windows: Fixed-size, non-overlapping intervals (e.g., every 5 minutes)
• Sliding windows: Fixed-size intervals that overlap, triggered at a defined slide interval
• Session windows: Dynamic intervals defined by a period of inactivity, ideal for grouping user behavior

Windowing is essential for computing metrics like peak load per minute or user session length in real-time dashboards.

A heuristic that tracks event-time progress in a stream processor. It declares that no events with a timestamp earlier than the watermark are expected to arrive.

• Handles out-of-order data caused by network latency or clock skew
• Determines when a window can be finalized and emitted
• A late-arriving threshold can be set to accept stragglers after the watermark passes

Without watermarks, a processor would wait indefinitely for late data, blocking downstream outputs. They are the backbone of exactly-once temporal determinism.

A fault-tolerance mechanism that periodically snapshots the entire state of a streaming operator to durable storage. On failure, the processor restores from the last successful checkpoint.

• Enables exactly-once semantics by coordinating state saves with source offsets
• Barrier-based checkpointing in Apache Flink injects markers into the stream to create a consistent global snapshot
• Incremental checkpoints reduce overhead by only saving state changes since the last snapshot

Checkpointing is what separates a toy streaming demo from a production-grade, crash-resilient pipeline.

A persistent or in-memory storage engine embedded within a stream processor to hold intermediate computation state. Stateful operations like joins and aggregations require a state store.

• RocksDB is a common embedded key-value store for local state in Apache Flink and Kafka Streams
• Remote state backends store state externally for faster recovery and elasticity
• State is partitioned by key to align with stream partitioning for local access

The state store is the memory of a stream processor, enabling it to correlate events across time without external database calls.

A delivery guarantee ensuring that each record is processed exactly once, even in the face of producer retries or consumer failures. This prevents both data duplication and data loss.

• Achieved through idempotent producers, transactional writes, and checkpoint coordination
• Requires the stream processor, source, and sink to all participate in a distributed transaction protocol
• Two-phase commit is a common implementation pattern for atomic writes to external sinks

Exactly-once semantics is critical for financial aggregations and inventory counts, where double-counting is unacceptable.

A flow control mechanism that prevents a fast upstream producer from overwhelming a slow downstream consumer. Without backpressure, buffers overflow and data is lost.

• TCP-based backpressure propagates naturally in systems like Akka Streams by blocking the receive buffer
• Credit-based flow control in Reactive Streams allows consumers to signal how many records they can accept
• In Apache Kafka, backpressure is implicit: consumers pull data at their own pace via offset commits

Proper backpressure handling ensures system stability under load spikes without manual throttling.