Stream Processing Engine

In financial fraud detection, duplicate or lost transactions are unacceptable. Stream processing engines must provide exactly-once semantics to ensure each transaction is processed precisely one time, even during worker failures or network partitions.

• Mechanism: Achieved through distributed checkpointing and idempotent writes to state stores.
• Fraud Relevance: Prevents double-counting in velocity checks, which would falsely inflate risk scores and block legitimate customers.
• Implementation: Often relies on a two-phase commit protocol between the engine and the sink, or leveraging transactional APIs in message brokers like Apache Kafka.

A method of tracking and analyzing streams of information about events to identify meaningful patterns, causal relationships, and composite events across multiple data sources in near real-time. CEP engines apply temporal pattern matching to detect sequences like 'withdrawal followed by rapid transfer to new beneficiary' within milliseconds.

• Detects event sequences, not just individual anomalies
• Uses event pattern languages (EPL) for declarative rule definition
• Complements ML models by encoding known fraud signatures
• Examples: Esper, Siddhi, Apache Flink CEP

A continuous computation technique that calculates metrics over a dynamically updating, overlapping time interval to provide rolling statistics on streaming data. In fraud detection, sliding windows compute velocity metrics like 'transactions per card in the last 5 minutes' that update with each new event.

• Hopping windows: fixed-size, overlapping intervals
• Sliding windows: continuous, triggered by each event
• Enables real-time velocity checks and behavioral baselines
• Requires efficient state management for high-cardinality keys

A distributed event streaming platform used as the central nervous system for building real-time data pipelines, decoupling producers and consumers of high-throughput transaction streams. Kafka provides durable, partitioned, replicated logs that enable exactly-once semantics and replayability for fraud model training.

• Handles millions of messages per second with P99 latency under 5ms
• Log compaction preserves latest state for reference data
• Integrates with Kafka Streams and KSQL for stream processing
• Foundation of the hot path in Lambda architectures

A data processing guarantee ensuring that each record is processed precisely one time, eliminating data loss or duplication even in the event of system failures. In financial fraud pipelines, exactly-once semantics prevent double-counting transactions that would corrupt velocity calculations and risk scores.

• Requires idempotent producers and transactional consumers
• Implemented via two-phase commits or idempotency keys
• Critical for financial reconciliation and audit trails
• Supported by Kafka transactions and Flink checkpointing

An operation that enriches a real-time event stream by joining it with a reference table or materialized view stored in a state store. For example, linking an incoming transaction to a merchant risk profile or a cardholder's historical behavior pattern stored in a feature store.

• Enables data enrichment without external service calls
• Uses Change Data Capture (CDC) to keep tables current
• Reduces latency compared to REST-based enrichment
• Requires careful state TTL management to control memory

A flow control mechanism that prevents a fast data producer from overwhelming a slower consumer by applying a feedback signal to slow down or buffer the incoming data stream. In fraud scoring, backpressure ensures that downstream ML inference services are not flooded during transaction spikes.

• Implemented via Reactive Streams or TCP flow control
• Prevents out-of-memory errors in processing nodes
• Works with circuit breaker patterns for graceful degradation
• Essential for maintaining P99 latency SLAs under load