Long Short-Term Memory (LSTM)

The foundational innovation of LSTM architecture. The CEC is a self-connected linear unit that enforces constant error flow through time, preventing gradients from decaying or exploding during backpropagation. In a standard RNN, multiplying gradients across many time steps causes them to vanish exponentially; the CEC's fixed weight of 1.0 acts as a highway for gradient flow. This allows the network to bridge time lags exceeding 1000 discrete steps, enabling it to learn dependencies between a market event and its delayed effect on price.

A sigmoid-activated neural layer that decides what information to discard from the cell state. The gate examines the previous hidden state and current input, outputting a value between 0 (completely forget) and 1 (completely retain) for each element of the cell state vector. In financial forecasting, this mechanism allows the network to automatically purge stale information—such as an outdated volatility regime—when a structural market break occurs. Without it, the cell state would saturate with irrelevant historical noise, degrading predictive accuracy.

A two-part mechanism controlling what new information is stored in the cell state:

• A sigmoid layer decides which values to update
• A tanh layer creates a vector of candidate values to add

The element-wise multiplication of these two outputs ensures only relevant new information is integrated. For a high-frequency trading model processing tick data, the input gate might selectively store a sudden order flow imbalance while ignoring routine noise, updating the cell state only when a statistically significant signal arrives.

Determines the hidden state output based on the filtered cell state. The gate applies a sigmoid function to the concatenated previous hidden state and current input, then multiplies this by a tanh-squashed version of the updated cell state. This two-stage filtering ensures the network only exposes task-relevant information at each time step. In a multi-horizon price prediction model, the output gate might suppress short-term microstructure noise when the training objective is forecasting the VWAP over a 15-minute horizon.

An architecture that runs two independent LSTM layers in opposite directions across the input sequence—one processing from past to future, the other from future to past—and concatenates their hidden states. This provides the output layer with full temporal context from both directions at every time step. In financial applications, a BiLSTM can condition a price prediction on both the preceding order flow and the subsequent market reaction, which is particularly valuable for non-causal tasks like anomaly detection or post-trade classification where the entire sequence is available.

An architectural variant where the gates are given direct access to the cell state. Instead of conditioning gate decisions solely on the hidden state and input, peephole connections feed the cell state vector directly into the forget, input, and output gates. This allows each gate to inspect the current memory before acting. For financial time series with precise timing requirements—such as detecting the exact moment a limit order book imbalance triggers a regime shift—peephole connections enable the network to learn more accurate temporal dependencies by letting the cell state directly influence its own gating.

The defining architectural innovation of the LSTM. Three gates—forget, input, and output—regulate information flow through the cell state using sigmoid activations and pointwise multiplication. The forget gate decides what to discard from the previous cell state, the input gate determines which new information to store, and the output gate controls what part of the cell state is exposed to the hidden state. This gating structure allows gradients to flow unchanged across hundreds of time steps, directly mitigating the vanishing gradient problem that cripples standard RNNs.

The horizontal highway running through the entire LSTM chain. The cell state (C_t) acts as a conveyor belt carrying long-term memory across the sequence with only minor linear interactions. Information can be added or removed via the gates, but the cell state itself undergoes minimal transformation at each step. This design enables the network to preserve signals from early time steps—such as a price pattern that formed 100 ticks ago—and carry them forward to influence predictions at the current step without degradation.

A fundamental obstacle in training deep recurrent networks. During backpropagation through time, gradients are multiplied repeatedly by weight matrices. If these weights are small (< 1), the gradient signal decays exponentially, preventing the network from learning dependencies beyond ~10 time steps. LSTMs solve this through the Constant Error Carousel—the cell state's additive update mechanism preserves gradient magnitude. This is why LSTMs can capture dependencies spanning hundreds of ticks in high-frequency order book data, while vanilla RNNs fail.

An extension that processes the input sequence in both forward and backward directions using two separate LSTM layers. The hidden states from both directions are concatenated at each time step, giving the output layer access to both past and future context simultaneously. In financial forecasting, this allows the model to condition a prediction at time t on both the preceding order flow imbalance and the subsequent market reaction—useful for non-causal tasks like volatility estimation or anomaly detection where the full sequence is available.

An LSTM variant where the gates are allowed to inspect the cell state directly. Standard LSTMs compute gate activations using only the current input and previous hidden state. Peephole connections add the previous cell state (C_{t-1}) as input to the forget and input gates, and the current cell state (C_t) to the output gate. This gives the network precise timing information—critical for tasks requiring exact interval measurement, such as detecting rhythmic patterns in tick-level microstructure noise or periodic liquidity events.

An encoder-decoder architecture where one LSTM encodes the input sequence into a fixed-length context vector, and a second LSTM decodes it into a target sequence. The encoder compresses the history of limit order book events into a latent representation; the decoder autoregressively generates future price trajectories. A key enhancement is the attention mechanism, which allows the decoder to dynamically weight encoder hidden states at each decoding step, bypassing the bottleneck of a single context vector and dramatically improving long-sequence forecasting accuracy.