Transformer Architecture

The core computational unit that allows the model to weigh the relevance of every other transaction in a sequence when encoding a specific transaction. Unlike recurrent models that process data step-by-step, self-attention computes a weighted representation of the entire sequence simultaneously.

• Query, Key, Value Vectors: Each input element projects to three vectors. The dot product of the Query with all Keys produces attention scores, which weight the Value vectors.
• Scaled Dot-Product: Scores are scaled down by the square root of the dimension to prevent vanishing gradients from large dot products in high dimensions.
• Contextual Embedding: The output for a transaction is a sum of all other transactions weighted by their relevance, directly capturing a suspicious transfer's relationship to a login event from 50 steps prior.

Instead of performing a single attention function, the model runs multiple self-attention operations in parallel. Each 'head' can learn to focus on a different representational subspace, capturing diverse relational patterns in transaction data.

• Diverse Focus: One head might attend to temporal proximity, another to merchant category similarity, and a third to transaction amount magnitude.
• Parallel Learning: Outputs from all heads are concatenated and linearly projected, allowing the model to jointly attend to information from different representation subspaces at different positions.
• Fraud Context: This enables the model to simultaneously link a current high-value wire transfer to a recent address change and a historically anomalous beneficiary account.

Since the Transformer contains no recurrence or convolution, it has no inherent sense of sequence order. Positional encoding injects information about the absolute or relative position of each transaction in the sequence.

• Sinusoidal Encoding: Fixed sinusoidal functions of different frequencies are added to the input embeddings. Each dimension of the positional encoding corresponds to a sinusoid.
• Learned Encoding: Alternatively, position-specific embeddings can be learned during training, treating position as a feature.
• Temporal Order: This ensures the model distinguishes between the sequence 'Login, Transfer, Logout' and 'Transfer, Login, Logout', preserving the critical temporal causality of user behavior.

Each attention layer is followed by a position-wise feed-forward network applied identically and independently to every position. This introduces non-linearity and increases model capacity.

• Two Linear Transformations: The network consists of two linear transformations with a non-linear activation function, such as ReLU or GELU, in between.
• Position-Wise Processing: The same parameters are applied to each sequence element, but the processing is independent, allowing the model to refine the attention output for each transaction individually.
• Pattern Synthesis: While attention mixes information across the sequence, the feed-forward network synthesizes the resulting contextual representation into a more abstract feature for downstream fraud classification.

A critical architectural wrapper around each sub-layer (attention and feed-forward) to stabilize training and enable the construction of very deep models.

• Residual Connection: The input to a sub-layer is added directly to its output. This provides a direct 'highway' for gradient flow during backpropagation, mitigating the vanishing gradient problem in deep architectures.
• Layer Normalization: Applied after the residual addition, this normalizes the activations across the feature dimension for each individual transaction, stabilizing the learning dynamics.
• Deep Stacking: This LayerNorm(x + Sublayer(x)) pattern allows Transformers to be stacked 12, 24, or even more layers deep, enabling the learning of highly hierarchical and abstract temporal fraud signatures.

In the decoder stack, self-attention is modified to prevent positions from attending to subsequent positions. This ensures that the prediction for a future transaction depends only on known past outputs.

• Causal Masking: A mask is applied to the attention scores, setting all values for illegal connections (future time steps) to negative infinity before the softmax, making their weight zero.
• Autoregressive Generation: This preserves the autoregressive property required for sequence generation, such as predicting the next likely merchant category in a normal behavioral sequence.
• Fraud Simulation: For anomaly detection, a masked decoder can be trained to predict the next event in a sequence; a deviation between the predicted and actual event signals a potential anomaly.

The computational core of the Transformer. Self-attention computes a weighted representation of every element in a sequence by comparing it against all other elements, allowing the model to dynamically focus on the most relevant past transactions when analyzing a current event. It calculates Query (Q), Key (K), and Value (V) vectors for each input, producing an output weighted by the compatibility of the query with all keys. This mechanism directly captures long-range dependencies without the sequential bottleneck of recurrent networks.

An extension of self-attention that runs multiple attention operations in parallel. Each 'head' learns to focus on different representational subspaces at different positions, enabling the model to jointly attend to information from various perspectives. For fraud detection, one head might focus on transaction amount patterns while another tracks merchant category sequences, and a third monitors temporal intervals between events. Outputs from all heads are concatenated and linearly projected.

Since the Transformer contains no recurrence or convolution, it has no inherent sense of token order. Positional encoding injects information about the absolute or relative position of each transaction in the sequence. This is achieved by adding sinusoidal functions of different frequencies to the input embeddings. For temporal fraud modeling, this preserves the critical chronological order of transactions, allowing the model to distinguish between a legitimate sequence and a reordered, suspicious one.

The original Transformer architecture consists of an encoder stack and a decoder stack. The encoder maps an input sequence to a continuous representation. The decoder generates an output sequence one element at a time, consuming the encoder's output and previously generated tokens. For sequence-to-sequence anomaly detection, an encoder can compress a transaction history into a latent vector, and a decoder attempts to reconstruct it; a high reconstruction error signals fraud.

An attention-based architecture specifically designed for multi-horizon time-series forecasting. TFT integrates three key data types: static metadata (e.g., account type), past observations (transaction history), and known future inputs (e.g., upcoming holidays). Crucially, it uses variable selection networks to suppress irrelevant inputs and provides interpretable feature importance via its attention layers, making it ideal for explainable financial predictions.

A structured state space sequence model offering a linear-time alternative to the Transformer's quadratic self-attention complexity. Mamba uses a selective scan mechanism to efficiently compress long-range context into a hidden state, filtering out irrelevant information. This makes it exceptionally well-suited for processing very long, high-frequency transaction streams where the computational cost of full attention becomes prohibitive.