Long Short-Term Memory (LSTM)

The core innovation of the LSTM architecture. The CEC is a linear self-loop that maintains a constant weight of 1.0, allowing error signals to flow backward across many time steps without decaying or exploding. This provides short-term memory storage that is protected from external perturbations by the gating units. Without the CEC, gradients would vanish exponentially as they are multiplied by weights at each time step during backpropagation.

The forget gate decides what information to discard from the cell state. It takes the previous hidden state (h_{t-1}) and current input (x_t), passes them through a sigmoid activation, and outputs a value between 0 and 1 for each number in the cell state.

• 0: Completely forget this information
• 1: Completely retain this information
• The gate enables the network to reset its memory when context shifts, such as when a user session ends or a new behavioral sequence begins

The input gate controls what new information is stored in the cell state. It operates in two stages:

1. A sigmoid layer (the input gate) decides which values to update
2. A tanh layer creates a vector of new candidate values (C̃_t) that could be added to the state

The combined output selectively writes new information into the memory cell, preventing irrelevant inputs from corrupting long-term dependencies. This is critical for session-based recommendation where only certain user actions should influence future predictions.

The output gate determines what part of the cell state is exposed as the hidden state output. It applies a sigmoid function to filter the cell state, then multiplies it by a tanh of the cell state to produce the final output.

This mechanism ensures that the network only reveals information relevant to the current prediction task while keeping other stored knowledge internal. In next-event prediction, this allows the model to output a probability distribution over possible actions based on selectively retrieved historical context.

The horizontal line running through the top of the LSTM cell acts as a gradient superhighway. Information flows along this path with only minor linear interactions, allowing gradients to propagate unchanged across hundreds of time steps.

Key properties:

• Additive updates: Gates add or remove information rather than multiplying the entire state
• Uninterrupted flow: Unlike standard RNNs where hidden states are fully recomputed, the cell state provides a direct path for long-range gradient propagation
• This architecture is what makes LSTMs effective for modeling dwell time patterns and extended user journeys

An architectural variant where the gates are allowed to inspect the actual cell state before making decisions. Standard LSTM gates only see the previous hidden state and current input; peephole connections feed the cell state directly into the forget, input, and output gates.

This is particularly useful for tasks requiring precise timing, such as temporal point processes where the exact duration between events matters. Peephole connections allow the network to learn when to reset its memory based on the content of the memory itself, improving performance on sequence generation tasks.

LSTMs process a user's clickstream sequence within a session to predict next-click probability or purchase intent. Unlike stateless models, the forget gate allows the network to discard irrelevant early browsing while the input gate retains signals indicating genuine product interest. This architecture is particularly effective for anonymous sessions where no historical user profile exists, relying purely on in-session behavioral trajectory to generate real-time recommendations.

LSTMs model long-term preference drift by connecting user behavior across multiple sessions separated by days or weeks. The cell state acts as a persistent memory highway, carrying forward stable preferences while the output gate modulates how past interests influence current recommendations. This enables retailers to distinguish between a temporary gift purchase and a genuine category interest shift, preventing stale personalization based on outdated session data.

LSTMs inherently learn adaptive time-decay functions through their gating mechanism, eliminating the need for manual recency weighting. Key behavioral features processed include:

• Dwell time on product detail pages
• Inter-event latency between clicks
• Scroll depth and media engagement signals
• Cart abandonment timing patterns

The network autonomously learns that a 30-second product view yesterday may be more predictive than a 2-second view an hour ago.

LSTMs identify subtle change points in user behavior that precede funnel abandonment. By training on sequences labeled with survival analysis outcomes, the model learns to recognize the behavioral signatures of users about to exit. This enables real-time intervention triggers—such as offering free shipping or live chat—at the precise moment when the probability of abandonment crosses a learned threshold, directly improving conversion rate optimization.

LSTMs address the item cold start problem by leveraging sequential context rather than item interaction history. When a new product enters the catalog, the model infers its relevance based on the behavioral sequence embedding preceding its appearance. If a user's recent browsing trajectory shows affinity for similar attributes, the LSTM can appropriately rank the new item without requiring prior click-through data, accelerating discovery for fresh inventory.

Production LSTM models for retail use Truncated Backpropagation Through Time to handle sessions with hundreds of events. Rather than unrolling the full sequence, gradients are computed over fixed windows of 20-50 time steps, with the hidden state carried forward. Combined with session-parallel mini-batches, this enables efficient training on GPU clusters while preserving the network's ability to learn dependencies spanning the entire user journey.

The fundamental training obstacle that LSTM was explicitly designed to overcome. In standard Recurrent Neural Networks (RNNs) , the gradient signal used to update weights decays exponentially as it is propagated backward through time. This prevents the network from learning dependencies between events separated by more than ~10 time steps. The LSTM's gating mechanism creates additive gradient highways, allowing the error signal to flow backward across hundreds or thousands of steps without vanishing.

The core innovation of the LSTM cell, consisting of three multiplicative gates that regulate information flow:

• Forget Gate: Decides what existing memory to discard using a sigmoid layer
• Input Gate: Selects which new information to store in the cell state
• Output Gate: Determines what part of the cell state to expose as the hidden output Each gate is a small neural network with sigmoid activation, outputting values between 0 and 1 to scale information flow element-wise.

A streamlined variant of the LSTM introduced by Cho et al. (2014) that merges the cell state and hidden state into a single vector. The GRU uses only two gates:

• Reset Gate: Controls how much past information to forget
• Update Gate: Balances between retaining old state and incorporating new input With fewer parameters than LSTM, GRUs often train faster and perform comparably on smaller datasets, though LSTMs maintain an edge on very long sequences requiring precise memory control.

An architecture that runs two independent LSTM layers in opposite directions over the input sequence and concatenates their hidden states. The forward LSTM processes the sequence left-to-right, while the backward LSTM processes it right-to-left. This gives each time step access to both past and future context simultaneously. BiLSTMs are particularly effective for tasks like named entity recognition and speech recognition where full sequence context is available at inference time.

The architecture that largely displaced LSTMs as the dominant sequence model after Vaswani et al. (2017) . Instead of recurrent processing, Transformers use self-attention to compute pairwise interactions between all positions in a sequence simultaneously. This enables:

• Fully parallelized training across sequence positions
• Direct modeling of dependencies regardless of distance
• Superior scaling to massive datasets However, the quadratic complexity of self-attention means LSTMs retain advantages for extremely long sequences and resource-constrained deployments.

An LSTM variant where the gating layers are allowed to inspect the cell state directly when making decisions. In the standard LSTM, gates see only the previous hidden state and current input. Peephole LSTMs add connections from the cell state to the forget, input, and output gates, giving each gate direct visibility into the memory cell. This modification improves performance on tasks requiring precise timing, such as learning to count or generate rhythmic patterns.