Stream Processor

Continuously consumes data from event sources like Apache Kafka, Amazon Kinesis, or MQTT brokers without a predefined end. Unlike batch systems that process finite datasets, stream processors handle infinite data streams by design.

• Ingests millions of events per second with sub-millisecond latency
• Supports multiple source connectors for databases, message queues, and IoT hubs
• Maintains exactly-once semantics to prevent duplicate processing during failures

Maintains fault-tolerant local state to perform computations that span multiple events, such as aggregations, joins, and pattern detection. Uses event time—the timestamp when an event actually occurred—rather than processing time to guarantee correct results even with out-of-order data.

• RocksDB-backed state stores persist operator state to local disk
• Handles late-arriving data through watermarking and allowed lateness policies
• Enables exactly-once state consistency via distributed checkpointing

Divides unbounded streams into finite windows for meaningful computation. Supports multiple window types to match business logic:

• Tumbling windows: Fixed-size, non-overlapping intervals (e.g., every 5 minutes)
• Sliding windows: Overlapping intervals for moving averages (e.g., last 30 seconds, updated every 5 seconds)
• Session windows: Dynamic windows based on activity gaps, critical for user sessionization
• Global windows: Single window for the entire stream, used with custom triggers

Treats streams and tables as two sides of the same coin. A stream represents a changelog of events over time, while a table represents the current state at a point in time. This duality enables:

• Stream-stream joins: Correlating events from two live sources (e.g., ad impressions with click events)
• Stream-table joins: Enriching events with reference data (e.g., user profile lookup)
• Table-table joins: Combining materialized views for multi-dimensional analysis
• Change Data Capture (CDC) integration to convert database tables into streams

Achieves resilience through asynchronous distributed snapshots based on the Chandy-Lamport algorithm. Periodically checkpoints the entire processing graph state to durable storage without pausing data ingestion.

• Lightweight barriers flow through the operator graph to mark consistent cut points
• On failure, restores state from the latest checkpoint and rewinds to the correct offset
• Enables zero data loss recovery in seconds, not minutes
• Checkpoints stored in distributed file systems like HDFS or S3 for durability

Implements reactive flow control to prevent overwhelming downstream operators or external sinks when processing spikes occur. The system dynamically throttles source consumption based on the slowest component in the pipeline.

• Uses credit-based flow control where consumers grant permits to producers
• Prevents out-of-memory errors by bounding in-flight data buffers
• Maintains end-to-end latency stability under variable load conditions
• Integrates with circuit breaker patterns for external service calls

A methodology for tracking and analyzing streams of data to infer complex patterns that signify critical opportunities or threats. Unlike simple stream filtering, CEP engines apply temporal pattern matching and causal event correlation to detect composite events.

• Detects sequences like 'item added to cart' followed by 'payment failure' within 5 minutes.
• Uses declarative rules or SQL-like languages for pattern definition.
• Drives immediate automated responses, such as triggering a fraud alert or a personalized discount offer.

The fundamental technique for dividing an unbounded data stream into finite, manageable chunks for stateful operations like aggregation and joining. Without windowing, a stream processor cannot compute a moving average or a session-based count.

• Tumbling windows: Fixed-size, non-overlapping intervals (e.g., every 5 seconds).
• Sliding windows: Fixed-size intervals that advance by a step smaller than the window size, creating overlap.
• Session windows: Dynamically sized windows that capture periods of activity separated by gaps of inactivity.

A critical resilience mechanism that allows a consumer to signal upstream producers that it is overwhelmed, preventing buffer bloat and out-of-memory crashes. A robust stream processor implements reactive pull-based backpressure.

• Prevents system failure under load spikes by controlling the flow of data.
• Implemented via protocols like Reactive Streams or TCP windowing.
• Contrasts with naive buffering, which only delays an inevitable crash.

A software architecture pattern that simplifies the Lambda Architecture by treating all data—both real-time and historical—as a single stream. A single stream processor engine handles both use cases, eliminating the need to maintain separate batch and speed layers.

• Reprocessing historical data is achieved by replaying the immutable event log.
• Dramatically reduces code duplication and operational complexity.
• Relies on a durable, replayable log like Apache Kafka as the source of truth.

An architectural pattern where the state of an application is determined by a sequence of immutable, append-only events. The stream processor consumes this event log to build materialized views and projections.

• Provides a complete, auditable source of truth for all state changes.
• Enables temporal queries to reconstruct application state at any point in time.
• Pairs naturally with CQRS to separate write-optimized event logs from read-optimized query models.

A processing strategy that simulates a continuous stream by collecting incoming records into small, discrete batches and processing them at very short intervals (e.g., every 100ms). This is the execution model of Spark Streaming.

• Trades off true sub-millisecond latency for higher throughput and simpler fault recovery.
• Contrasts with true event-at-a-time processing, which yields lower latency.
• A pragmatic choice when end-to-end latency requirements are in the seconds, not milliseconds.