Lambda Architecture

The foundational principle of Lambda Architecture is the append-only, immutable data store. Raw transaction events are never overwritten or deleted; new data is simply appended as it arrives. This guarantees an auditable system of record for financial fraud investigations.

• Event Sourcing: Every state change is captured as an event
• Temporal Querying: Enables point-in-time reconstruction of account states
• Correction by Addition: Errors are fixed by writing new compensating records, not altering history

The batch layer precomputes comprehensive views by periodically re-processing the entire master dataset using distributed processing frameworks. It prioritizes accuracy and completeness over latency, generating immutable, indexed batch views that serve as the ground truth for fraud models.

• Runs computationally intensive algorithms like graph neural network training on full transaction histories
• Corrects any errors introduced by the speed layer's approximations
• Typical latency: minutes to hours; output is written to a serving layer

The speed layer compensates for the high latency of batch updates by processing only the most recent data incrementally. It uses stream processing engines and approximate algorithms to generate real-time views, enabling sub-second fraud risk scoring on transactions that arrived after the last batch computation.

• Employs techniques like Count-Min Sketch and Bloom Filters for memory-efficient frequency estimation
• Sacrifices some accuracy for extreme speed; results are inherently temporary
• Continuously discards data once the batch layer catches up and computes the authoritative view

The serving layer indexes the precomputed batch views and exposes them for low-latency random reads. A query for a complete result is satisfied by merging the stale-but-accurate batch view with the fresh-but-approximate real-time view. This provides a unified, queryable interface for the risk scoring engine.

• Stores indexed batch views in a distributed database optimized for random access
• Merges results from batch and speed layers transparently to the application
• Ensures the authorization flow receives a complete risk profile without coupling to the underlying complexity

A core tenet of Lambda Architecture is that systems must be resilient to human error, which is considered more dangerous than machine failure. The immutable master dataset and recomputation-based batch layer ensure that catastrophic bugs in application logic or accidental data deletions are fully recoverable.

• Recomputation: If a fraud model is found to have a logic error, the batch layer can simply recompute all historical views from the raw, uncorrupted master data
• Eliminates the need for complex and risky rollback procedures in online databases
• Provides a robust safety net for iterative model development and debugging

The primary criticism of Lambda Architecture is the operational burden of maintaining two separate processing systems—batch and speed—that must produce logically equivalent results. This often requires maintaining two distinct codebases and merging their outputs, which can introduce subtle semantic inconsistencies.

• Code Duplication: Transformation logic must be written for both a batch framework (e.g., Apache Spark) and a stream processor (e.g., Apache Flink)
• Operational Overhead: Requires expertise in managing and tuning two distinct distributed systems
• This complexity motivated the evolution toward Kappa Architecture, which eliminates the batch layer entirely by treating all data as a stream

The immutable, append-only historical data store responsible for computing accurate but high-latency views. In fraud detection, this layer processes terabytes of settled transactions using distributed processing frameworks like Apache Spark or Hadoop MapReduce to train models on complete datasets. It serves as the system of record, periodically recomputing batch views that correct approximations made by the speed layer. Key characteristics include:

• Stores the master dataset in a distributed file system
• Recomputes batch views on a scheduled cadence (e.g., hourly or nightly)
• Provides eventual accuracy by correcting real-time approximations
• Handles complex feature engineering that requires full historical context

The real-time stream processing component that compensates for the batch layer's high latency. It ingests live transaction events and applies incremental algorithms to produce low-latency views within milliseconds. In fraud pipelines, this layer executes online inference against pre-trained models, performs sliding window aggregations for velocity checks, and scores transactions before authorization completes. Design considerations include:

• Uses stream processors like Apache Kafka Streams or Apache Flink
• Maintains state in-memory or in fast key-value stores for sub-millisecond access
• Handles out-of-order events using watermarks and allowed lateness
• Results are approximate and later reconciled against batch views

The query-facing merge point that combines batch and real-time views into a unified response. It indexes batch-computed results and merges them with incremental speed layer updates to provide a complete, up-to-date picture for each transaction. In fraud systems, this layer enables the risk scoring engine to query enriched features—such as a cardholder's 30-day spending profile from the batch layer combined with the last 5 minutes of velocity from the speed layer—in a single call. Implementation patterns include:

• Read-optimized databases like Apache HBase or Cassandra for batch view serving
• Merge-on-read logic that combines batch and real-time results at query time
• Caching strategies to reduce latency on frequently accessed entity profiles

An alternative to Lambda that eliminates the separate batch layer by treating all data as a stream. Instead of maintaining dual processing paths, Kappa uses a single stream processing engine to handle both real-time and historical computations by replaying events from a durable log like Apache Kafka. This simplifies operational complexity at the cost of requiring the stream processor to handle full historical reprocessing. Trade-offs for fraud detection include:

• Reduced code duplication—single codebase for all processing
• Simpler operations—no batch scheduling or dual-path maintenance
• Higher replay latency—reprocessing years of transactions through a stream is slower than batch
• Best suited when event sourcing is the primary data pattern and batch-specific optimizations are unnecessary

The conceptual division of data processing responsibilities within Lambda. The hot path (speed layer) handles time-critical data requiring sub-second latency—blocking a transaction, scoring risk, triggering alerts. The cold path (batch layer) processes data where accuracy matters more than speed—training models, generating regulatory reports, detecting slow-burn fraud patterns. This separation allows each path to be optimized independently:

• Hot path: In-memory state, approximate algorithms, strict latency SLAs with circuit breakers
• Cold path: Disk-based storage, exact computations, fault-tolerant with retry logic
• Reconciliation: Cold path results periodically overwrite hot path approximations
• In fraud systems, the hot path might use a Count-Min Sketch for approximate velocity while the cold path computes exact counts

A critical enrichment operation where a live transaction event stream is joined against a reference table or materialized view. In Lambda, the batch layer maintains the ground-truth table (e.g., merchant risk profiles, cardholder behavioral segments) while the speed layer performs the join at inference time. This enables the risk scoring engine to combine real-time signals with historical context. Implementation considerations:

• Uses Change Data Capture to keep reference tables synchronized
• Employs state stores in stream processors for local, low-latency lookups
• Handles temporal joins where the reference data version at event time must be used
• Critical for data enrichment steps that add device fingerprints, geolocation, and watchlist status