Positional Encoding

The original method from Attention Is All You Need uses sine and cosine functions of varying frequencies to encode position. Each dimension of the positional vector corresponds to a sinusoid.

• Formula: PE(pos, 2i) = sin(pos / 10000^(2i/d_model))
• Key Property: Allows the model to easily learn to attend to relative positions, as PE(pos+k) can be represented as a linear function of PE(pos).
• Benefit: No additional parameters to learn and can theoretically extrapolate to sequence lengths longer than those seen during training.

An alternative approach where position representations are treated as learnable parameters, initialized randomly and updated via backpropagation. Common in BERT and GPT architectures.

• Mechanism: A standard embedding layer maps integer indices [0, 1, ..., N] to dense vectors.
• Constraint: The maximum sequence length is fixed at training time; the model cannot natively handle longer sequences without interpolation.
• Use Case: Often preferred when the training data distribution of sequence lengths is well-defined and bounded, such as processing a fixed window of 100 prior transactions.

Instead of encoding absolute position, this method modifies the self-attention mechanism to depend on the pairwise distance between tokens. This directly models the intuition that the relationship between transaction t and t-1 is more important than the absolute index of t.

• Implementation: Often achieved by adding a learned bias to the attention score based on the offset between query and key positions.
• Advantage: Naturally generalizes to sequence lengths unseen during training, making it robust for variable-length transaction histories.
• Example: Transformer-XL and T5 relative position biases.

A widely adopted method that encodes position by rotating the query and key vectors in the attention mechanism. The dot-product attention score then depends only on the relative distance between tokens.

• Mechanism: Multiplies vectors by a rotation matrix where the rotation angle is a function of the position index.
• Key Benefit: Seamlessly integrates absolute position information while naturally decaying the attention weight with increasing relative distance, providing a strong inductive bias for temporal locality.
• Relevance: Used in Llama, Mistral, and other state-of-the-art models for processing long sequences.

A simple yet effective method that adds a static, non-learned bias to the attention scores. The bias is a linear penalty proportional to the distance between tokens.

• Formula: A constant slope m is subtracted from the attention score for each step of separation.
• Advantage: Extremely computationally efficient with no learned parameters and has demonstrated strong extrapolation to sequences 2-3x longer than training length.
• Application: Ideal for high-throughput fraud detection pipelines where inference speed on long transaction histories is critical.

An approach that maps continuous time stamps directly to vectors, moving beyond discrete sequence positions to model the actual inter-arrival times between transactions.

• Method: Uses functions like cos(ω * t) or learnable neural networks to encode the real-valued time t of a transaction.
• Significance: Captures the temporal density of events—a burst of 10 transactions in 1 minute is fundamentally different from 10 transactions over 10 hours, a distinction lost by discrete positional indexing.
• Integration: Often concatenated with or added to the standard positional encoding to provide both sequential order and temporal rhythm information.

The foundational architecture that necessitates positional encoding. Unlike RNNs, Transformers process all tokens in parallel via self-attention, making them inherently permutation-invariant. Positional encoding is the mechanism that injects sequence order, allowing the model to distinguish 'Transaction A before B' from 'B before A'. This enables the capture of long-range dependencies without recurrent bottlenecks.

A recurrent architecture that processes sequences step-by-step, inherently encoding position through its recurrent state. Unlike Transformers, LSTMs do not require explicit positional encoding because their computational graph is unrolled over time. However, this sequential nature prevents parallelization and can struggle with very long-range dependencies due to vanishing gradients, motivating the shift to attention-based models for large transaction logs.

Uses causal, dilated convolutions to model sequences. The causal constraint ensures no information leakage from the future, while dilation expands the receptive field. Positional information is implicit in the convolutional structure, as filters slide across the time dimension. TCNs offer a parallelizable alternative to RNNs and a simpler positional mechanism than Transformers, often serving as a robust baseline for sequence modeling.

The process of mapping a variable-length sequence of discrete events (e.g., merchant codes) into a fixed-length vector. Effective sequence embeddings must preserve temporal order, often achieved by feeding sequences into a position-aware model like a Transformer or LSTM. The resulting vector captures the semantic and temporal essence of a user's behavior, enabling comparison, clustering, and anomaly detection on entire transaction histories.