Kappa Architecture

The foundational principle of Kappa Architecture is the use of a single technology stack for all data processing workloads. Unlike Lambda Architecture, which maintains separate batch and speed layers, Kappa treats all data as a stream.

• Real-time and historical data are processed by the same engine (e.g., Apache Flink, Kafka Streams)
• Eliminates the operational burden of maintaining two distinct codebases
• Simplifies the system architecture to a single, unified processing graph
• Reprocessing historical data is achieved by replaying the stream from a durable log

Kappa Architecture relies on a distributed, durable, and immutable log (typically Apache Kafka) as the central source of truth. All data entering the system is written as an ordered sequence of events.

• The log retains the full history of events, configurable via retention policies
• Enables event sourcing, where the current state is derived by replaying the log
• Provides natural auditability and the ability to reconstruct any past state
• Decouples producers from consumers, allowing independent scaling

Batch reprocessing in Kappa is not a separate system but a function of replaying the event log through an updated version of the stream processor. When business logic or models change, the new code is applied to the historical stream.

• A new consumer group reads from the beginning of the topic (offset 0)
• The updated processor recomputes results and writes to a new output table or topic
• Once caught up, a cutover switches production traffic to the new results
• This provides the same recomputation capability as a batch layer without a separate codebase

Kappa architectures depend on stream processors that can maintain fault-tolerant, local state. Operations like aggregations, joins, and windowing require the engine to remember intermediate results.

• State is stored locally in RocksDB or similar embedded databases and backed up to a changelog topic
• Checkpointing periodically snapshots state to durable storage for recovery
• Enables exactly-once semantics even across failures
• State size can grow large; requires careful capacity planning and state TTL strategies

Real-world event streams are rarely perfectly ordered. Kappa Architecture embraces this by using event time semantics and watermarks to manage lateness deterministically.

• Events carry their own timestamp, independent of processing time
• Watermarks are heuristics that declare how late the system will wait for stragglers
• Late-arriving events beyond the watermark can be routed to a side output or dead letter queue
• This ensures results are both timely and eventually accurate without blocking the pipeline

The primary motivation for Kappa is reducing operational complexity. Lambda Architecture requires maintaining and synchronizing two separate systems—a batch layer (e.g., Spark) and a speed layer (e.g., Storm).

• Kappa eliminates the need to write and debug the same logic twice in different languages
• Monitoring, deployment, and scaling are unified under one framework
• However, Kappa shifts complexity to the stream processor's ability to handle large historical replays efficiently
• Best suited for use cases where the event log is the natural source of truth and batch-only legacy systems are absent

An architectural pattern where the state of an application is determined by a sequence of immutable events. This aligns perfectly with Kappa Architecture, as the event log becomes the single source of truth, allowing state to be rebuilt at any point by replaying the stream.

• State is derived, not stored
• Provides a full audit trail
• Enables temporal queries and debugging

A critical delivery guarantee in distributed stream processing ensuring that each record is processed once and only once, even during failures. Kappa architectures rely on this to maintain accurate state and avoid duplicate side effects.

• Achieved via idempotent producers and transactional sinks
• Requires coordinated checkpointing and recovery
• Essential for financial calculations and inventory counts

A mechanism that tracks event-time progress in a stream processor. Watermarks declare that no more data with a timestamp earlier than T is expected, allowing windowed operations to finalize results. This is essential for handling out-of-order data in a Kappa pipeline.

• Triggers the closing of time windows
• Balances result completeness against latency
• Prevents infinite waiting for late-arriving data