Transformer

The core computational unit that allows the model to weigh the importance of every element in a sequence when encoding a specific element. It computes attention scores by taking the dot product of a Query vector with all Key vectors, scaling the result by the square root of the dimension (√d_k) to prevent vanishing gradients in the softmax. This operation is executed in parallel across the entire sequence, enabling the capture of long-range dependencies without the sequential bottleneck of recurrent neural networks. The final output is a weighted sum of the Value vectors.

Instead of performing a single attention function, the model linearly projects the Queries, Keys, and Values h times with different, learned linear projections. This allows the model to jointly attend to information from different representation subspaces at different positions. For example, one head might focus on syntactic dependencies while another tracks long-range semantic relationships. The outputs of all heads are concatenated and projected once more, yielding a richer representation than a single attention head could provide.

Since the Transformer contains no recurrence or convolution, it has no inherent sense of token order. Positional encodings are injected at the bottom of the encoder and decoder stacks to inject information about the relative or absolute position of tokens. The original architecture uses sinusoidal functions of varying frequencies: PE(pos, 2i) = sin(pos / 10000^(2i/d_model)) and PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model)). This allows the model to easily learn to attend by relative position and extrapolate to sequence lengths unseen during training.

Each encoder and decoder layer contains a fully connected feed-forward network applied identically to each position. This consists of two linear transformations with a ReLU activation in between: FFN(x) = max(0, xW₁ + b₁)W₂ + b₂. While the attention layers mix information across positions, the feed-forward layers process the representation at each position independently, adding a non-linear transformation that increases the model's capacity. The inner dimension is typically four times the model dimension (d_ff = 2048).

Each sub-layer (self-attention and feed-forward) is wrapped with a residual connection, where the input is added to the sub-layer's output: LayerNorm(x + Sublayer(x)). This mitigates the vanishing gradient problem in deep architectures, allowing the model to be stacked 6 or more layers deep. Layer normalization is applied after the addition, stabilizing the hidden state dynamics by normalizing across the feature dimension for each individual token, rather than across the batch dimension as in batch normalization.

To preserve the auto-regressive property during generation, the decoder's self-attention sub-layer is modified with a causal mask. This mask sets all attention scores for illegal future connections to negative infinity before the softmax, ensuring that the prediction for position i can depend only on known outputs at positions less than i. This is implemented by masking out (setting to -∞) the upper triangular portion of the attention score matrix, preventing leftward information flow and enforcing the sequential generation constraint.

The raw data substrate for Transformer-based HFT models. A Limit Order Book is an electronic record of all outstanding buy and sell orders for a specific asset, organized by price level and continuously updated in real-time.

• Transformers ingest LOB data as a sequence of snapshots or event-level updates
• The self-attention mechanism can learn to relate a new bid at level 5 to a cancellation at level 1 across a 100-millisecond window
• Typical LOB features include price, volume, and order count for the top 10-50 levels on both sides
• Pre-training on LOB reconstruction is a common self-supervised objective before fine-tuning for return prediction

The primary architectural competitor to the Transformer for time-series forecasting. A Temporal Convolutional Network uses dilated, causal convolutions to model sequential data with a flexible receptive field.

• Unlike Transformers, TCNs process data with causal padding to prevent information leakage from future timesteps
• Dilated convolutions allow an exponentially large receptive field with fewer layers
• TCNs offer deterministic inference latency, unlike the variable-length attention computation in Transformers
• Often benchmarked against Transformers on LOB prediction tasks; TCNs sometimes win on very short horizons (< 10 ticks)

A specialized Transformer variant that segments a time series into subseries-level patches rather than attending to individual timesteps. This design captures local semantic information efficiently.

• Each patch is a vector of consecutive timesteps, projected into the embedding space
• Channel-independence means each univariate series is processed by a shared Transformer backbone, preventing cross-channel noise
• Achieved state-of-the-art results on long-sequence forecasting benchmarks, outperforming standard Transformers
• The patching mechanism reduces the sequence length from L to L/P, where P is the patch size, cutting the O(n²) attention cost dramatically

The gold-standard evaluation framework for Transformer-based trading models. Walk-forward validation sequentially retrains a model on an expanding or rolling window of historical data and tests it on the immediately subsequent period.

• Prevents look-ahead bias by ensuring no future information contaminates training
• A typical split: train on 2018-2022, validate on H1 2023, test on H2 2023, then walk forward
• Transformers are particularly sensitive to temporal data leakage due to their global receptive field
• Purged K-Fold cross-validation is a complementary technique that removes overlapping observations and inserts an embargo period between training and testing sets

An alternative data sampling technique that creates bars not by fixed time or volume intervals, but when the amount of new information arriving in the market reaches a threshold.

• Tick imbalance bars sample when the signed tick count exceeds a threshold, capturing order flow toxicity
• Volume imbalance bars sample when the cumulative volume delta crosses a threshold
• Dollar bars sample when a fixed dollar amount has been traded, normalizing for corporate actions and inflation
• Transformers trained on information-driven bars often learn more robust attention patterns because the input sequence is aligned with market information arrival, not arbitrary clock time

A diagnostic technique for detecting distribution shift between training and test sets before deploying a Transformer model. It works by training a classifier to distinguish between the two datasets.

• If a simple classifier achieves an AUC > 0.7, the distributions are significantly different
• For Transformers, this often reveals that market regimes have shifted (e.g., from low-vol to high-vol)
• The technique can identify the specific features most responsible for the shift by analyzing feature importance
• Mitigation strategies include regime-switching models, adversarial training, or simply discarding the most non-stationary features