Self-Attention

Unlike recurrent models that process tokens sequentially, self-attention computes a weighted representation for every position in the sequence simultaneously. This is achieved by calculating attention scores between all pairs of elements in a single matrix multiplication.

• Mechanism: For a sequence of length n, it computes an n x n attention matrix.
• Benefit: Drastically reduces training time by leveraging GPU parallelism.
• Fraud Context: Allows a model to instantly compare a current transaction against every historical transaction in a user's session without sequential bottlenecks.

Each input element is projected into three distinct vectors: a Query (Q), a Key (K), and a Value (V). The attention weight is computed as the scaled dot-product of the Query with all Keys, determining how much focus to place on each corresponding Value.

• Formula: Attention(Q,K,V) = softmax(QK^T / √d_k)V
• Purpose: The Q-K dot product measures relevance, while the V vector carries the actual information to be aggregated.
• Fraud Context: A transaction's Query can dynamically seek out past transactions (Keys) with similar anomalous patterns, regardless of their temporal distance.

Instead of a single attention function, the model runs multiple attention operations in parallel using different, learned linear projections for Q, K, and V. This allows the model to attend to information from different representation subspaces at different positions.

• Structure: Typically 8 to 16 parallel attention heads.
• Benefit: One head might focus on transaction amount, another on merchant category, and a third on temporal proximity.
• Fraud Context: Enables simultaneous detection of multiple fraud indicators—like velocity spikes and unusual geolocation—within a single layer.

Because self-attention is permutation-invariant (it ignores element order), positional information must be explicitly injected into the input embeddings. This is done by adding sinusoidal functions or learned embeddings that represent the absolute or relative position of each transaction in the sequence.

• Method: Sine and cosine functions of different frequencies are added to the input vectors.
• Purpose: Allows the model to distinguish between a sequence of [Deposit, Withdrawal, Withdrawal] and [Withdrawal, Deposit, Withdrawal].
• Fraud Context: Preserves the critical temporal order of events, ensuring the model recognizes that a login event preceding a transfer is different from a transfer preceding a login.

The raw attention scores (dot products of Q and K) are scaled down by a factor of 1 / √d_k, where d_k is the dimension of the key vectors. This prevents the softmax function from entering regions with extremely small gradients when the dot products grow large in magnitude.

• Stabilization: Keeps the variance of the dot products constant.
• Result: Ensures stable training and a smoother attention distribution.
• Fraud Context: Prevents the model from collapsing to a state where it only attends to a single outlier transaction, maintaining a balanced view of the entire sequence history.

Each sub-layer (attention and feed-forward) is wrapped with a residual connection followed by layer normalization. The output is LayerNorm(x + Sublayer(x)). This architecture facilitates gradient flow through deep networks and stabilizes the learning process.

• Residuals: Allow the model to bypass a sub-layer if it is not useful, acting as identity shortcuts.
• Normalization: Standardizes the inputs across features, speeding up convergence.
• Fraud Context: Enables the construction of very deep Transformer models that can capture highly complex, non-linear fraud patterns without suffering from vanishing or exploding gradients.

The foundational deep learning architecture that relies entirely on self-attention to process sequences in parallel. Unlike recurrent models, it computes representations for all positions simultaneously, enabling the capture of long-range dependencies between any two transactions regardless of their distance in the sequence. In fraud detection, this allows the model to directly relate a current transaction to a suspicious event that occurred hundreds of steps earlier without the degradation caused by vanishing gradients.

• Eliminates recurrence for full parallelization
• Multi-head attention captures diverse relational patterns
• Layer normalization and residual connections stabilize training

A critical companion to self-attention that injects information about absolute or relative position of tokens in a sequence. Since self-attention is permutation-invariant by design, positional encoding is the sole mechanism preserving temporal order of transactions. Without it, the model cannot distinguish between a sequence of [deposit, withdrawal, transfer] and [transfer, withdrawal, deposit]. Common implementations include sinusoidal functions and learned embeddings.

• Preserves transaction chronology
• Enables relative distance computation between events
• Can be fixed or learned during training

An unsupervised architecture that compresses a variable-length transaction history into a fixed-length latent vector using an encoder, then reconstructs the sequence with a decoder. The reconstruction error serves as an anomaly score—normal behavior patterns are reconstructed accurately, while fraudulent sequences produce high error. Modern implementations often replace recurrent encoders with self-attention layers to better capture long-range transactional dependencies.

• Unsupervised training requires no labeled fraud data
• Latent vector captures compressed behavioral profile
• High reconstruction error signals anomalous sequences