Long Short-Term Memory (LSTM)

The foundational innovation of LSTM is the cell state, a linear highway of information flow that runs through the entire sequence chain. This CEC mechanism allows gradients to flow backward in time unchanged, directly addressing the vanishing gradient problem that plagues standard RNNs. By ensuring the error signal remains constant, the network can bridge time lags exceeding 1000 discrete time steps, making it capable of linking a fraudulent transaction today to a subtle account change made weeks prior.

Information flow into, out of, and within the cell state is regulated by three neural network gates, each a sigmoid layer outputting a value between 0 and 1:

• Forget Gate: Decides what information to discard from the cell state (e.g., forgetting an old, compromised device ID).
• Input Gate: Decides which new information to store in the cell state (e.g., adding a new high-value beneficiary).
• Output Gate: Decides what information from the cell state to output as the hidden state for the current time step (e.g., exposing a risk score based on current memory).

Unlike a simple RNN which only has a single hidden state, an LSTM maintains two distinct state vectors at each time step:

• Cell State (c_t): The long-term memory, carrying structural information across the entire sequence.
• Hidden State (h_t): The short-term working memory, representing the output for the current time step and serving as the input to the next. This separation allows the network to maintain a stable long-term context while dynamically computing a transient output, crucial for distinguishing a normal spending spree from a structured fraud layering scheme.

The LSTM solves the vanishing gradient problem through an additive update mechanism rather than a multiplicative one. In a standard RNN, the hidden state is overwritten completely at each step, causing gradients to be scaled repeatedly. In an LSTM, the cell state is updated by adding new information via the input gate and selectively forgetting old information via the forget gate. This additive interaction creates a direct, uninterrupted path for gradients during Backpropagation Through Time (BPTT), allowing the model to learn dependencies across thousands of transactions.

A common architectural variant includes peephole connections, which allow the gates themselves to inspect the cell state before making a decision. In this configuration, the forget, input, and output gates receive not just the current input and previous hidden state, but also the current cell state as input. This gives the gates a precise, memory-aware control mechanism, allowing the forget gate to reset the cell state at the exact moment a user's behavioral pattern fundamentally shifts, such as after a confirmed account takeover.

LSTMs are frequently deployed in a Sequence-to-Sequence Autoencoder (Seq2Seq AE) configuration for unsupervised fraud detection. An encoder LSTM compresses a variable-length transaction history into a fixed-length context vector, and a decoder LSTM attempts to reconstruct the original sequence from this vector. A high sequence anomaly score, derived from the reconstruction error, signals that a user's current session deviates significantly from their historical norm, indicating potential fraud without requiring labeled examples.

A streamlined alternative to LSTM that uses a reset gate and update gate to control information flow without a separate cell state. GRUs are computationally more efficient due to fewer parameters, making them well-suited for real-time fraud scoring where latency is critical. The update gate decides how much past information to retain, while the reset gate determines how to combine new input with previous memory. In practice, GRUs often match LSTM performance on shorter transaction sequences while training faster.

The training algorithm that makes LSTM learning possible. BPTT unrolls the recurrent network across time steps into a feedforward computation graph, then applies standard backpropagation. Key considerations:

• Truncated BPTT limits the unroll length to manage memory
• Gradients flow backward through both layers and time steps
• The LSTM's gating mechanism is specifically designed to prevent gradient decay during this process Without BPTT, the network cannot learn which historical transactions are predictive of future fraud.

The fundamental training obstacle that LSTMs were designed to solve. In standard RNNs, gradients shrink exponentially as they propagate backward through time, preventing the model from learning dependencies beyond ~10 time steps. The LSTM's cell state acts as a gradient highway, allowing error signals to flow unchanged across hundreds of steps. This is critical in fraud detection where a legitimate user's behavior pattern may span weeks of transactions before a deviation occurs.

A non-recurrent alternative that processes entire sequences in parallel using self-attention instead of sequential recurrence. Transformers capture dependencies between any two positions regardless of distance, eliminating the vanishing gradient issue entirely. For fraud detection, this means a transaction at time t=1 can directly influence the representation at t=500. The trade-off: quadratic memory complexity with sequence length, requiring careful engineering for long transaction histories.

Uses causal, dilated convolutions to model sequences with parallel computation. Key properties:

• Causality: output at time t depends only on inputs ≤ t
• Dilated convolutions: exponentially expand the receptive field
• Residual connections: stabilize training across many layers TCNs offer flexible receptive field sizing and avoid the sequential bottleneck of LSTMs, enabling faster training on GPU hardware while maintaining strict temporal ordering for transaction streams.

A structured state space sequence model offering linear-time sequence processing as an alternative to both LSTMs and Transformers. Mamba uses a selective scan mechanism that compresses context into a hidden state while dynamically filtering irrelevant information. For financial applications, this enables efficient processing of extremely long transaction histories (100k+ events) without the quadratic cost of attention or the sequential bottleneck of recurrence.