Apache Kafka

Kafka is fundamentally a distributed commit log where messages are persisted to disk and replicated across brokers. This architecture provides durability and fault tolerance—if a broker fails, other brokers with replicated partitions seamlessly take over. Unlike traditional message queues, Kafka retains messages for a configurable retention period, allowing consumers to reprocess historical transaction streams for model retraining or forensic analysis. The log’s immutable, append-only structure ensures strict ordering of events within a partition, which is critical for reconstructing the exact sequence of financial transactions.

Kafka implements a publish-subscribe pattern through topics. Producers write transaction events to specific topics, and multiple consumer groups independently read from them. This decoupling is vital for fraud pipelines: the same raw transaction stream can be consumed simultaneously by a real-time scoring engine, a long-term model training batch job, and an audit logging service without any consumer blocking another. Each consumer group maintains its own offset, allowing independent replay and processing speeds.

Kafka scales linearly by distributing topic partitions across multiple brokers. A high-volume topic like payment.transactions can be split into hundreds of partitions, allowing parallel consumption by multiple instances of a fraud scoring service. This partitioning model enables the system to handle the massive throughput of global payment networks. Adding new brokers triggers automatic partition rebalancing, scaling capacity without downtime. Partition keys ensure that all events for a specific entity (e.g., a credit card number) are routed to the same partition, preserving order for stateful velocity checks.

Kafka Streams is a client library for building real-time applications and microservices where input and output data are stored in Kafka clusters. It enables stateful operations like sliding window aggregations, joins, and transformations directly within the streaming layer. For fraud detection, this allows engineers to compute velocity checks (e.g., counting transactions per card in a 5-minute window) or enrich events by joining a transaction stream with a static table of known fraudulent devices, all without deploying a separate cluster like Apache Flink or Spark.

Kafka supports exactly-once semantics (EOS) to prevent duplicate processing, a critical requirement in financial systems where double-counting a transaction could falsely flag a legitimate customer or miss a fraudulent one. EOS guarantees that each record is processed precisely once, even across retries and broker failures. This is achieved through idempotent producers and transactional APIs that atomically write to multiple topics and commit consumer offsets, ensuring end-to-end consistency in the fraud scoring pipeline.

Kafka achieves sub-10ms latency from production to consumption by using zero-copy data transfer and batching optimizations. Producers can batch messages to reduce network round-trips, while consumers fetch large chunks of data sequentially from disk. For real-time fraud detection, this low latency ensures that a transaction event is available for scoring within milliseconds of being published, enabling the authorization flow to meet strict P99 latency budgets before the transaction is approved or declined.

The process of generating model predictions against a single transaction as it arrives, within a strict latency budget typically under 50ms. This requires pre-loaded models in memory and pre-computed features.

• Synchronous execution: The authorization flow blocks until a score is returned
• Model serving infrastructure: REST endpoints or gRPC services colocated with the stream processor
• Cold start avoidance: Models and feature caches must be pre-warmed

Online inference contrasts with batch scoring, where predictions are generated for large datasets on a scheduled basis without real-time constraints.

A continuous computation technique that calculates metrics over a dynamically updating time interval. Unlike tumbling windows with fixed boundaries, sliding windows overlap and update with each new event.

• Window size: The total lookback period, e.g., 1 hour
• Slide interval: How frequently the window advances, e.g., 1 minute
• Common metrics: Count, sum, average, distinct count

For fraud detection, sliding windows compute velocity checks such as 'number of transactions from this IP in the last 30 minutes' or 'distinct cards used at this terminal in the last hour.'

A fraud detection technique that monitors the frequency of specific attributes over short time windows to identify anomalous bursts of activity. High velocity is a strong indicator of automated attacks or card testing.

• Card velocity: Number of transactions per card within a time window
• IP velocity: Distinct cards or accounts accessed from a single IP
• Device velocity: Accounts accessed from a single device fingerprint
• Geographic velocity: Distance between consecutive transaction locations

Velocity thresholds are typically implemented as deterministic rules in a rules engine, while machine learning models consume velocity features as predictive signals.