Encoder-Decoder Architecture

The fundamental capability of transforming an input sequence of arbitrary length into an output sequence of arbitrary length. In demand forecasting, the encoder ingests historical sales data, promotional calendars, and pricing events, while the decoder generates future demand trajectories. This architecture handles the temporal misalignment between past observations and future predictions, unlike fixed-window models that require identical input-output dimensions.

The encoder compresses the entire input sequence into a fixed-length context vector (also called a thought vector or latent representation). This vector captures the essential features of the input: trend components, seasonal patterns, and recent shocks. The quality of this compression directly determines forecast accuracy. Information bottlenecks occur when the context vector cannot retain long-range dependencies, motivating attention mechanisms.

The decoder generates outputs one step at a time, using its own previous predictions as inputs for the next step. This autoregressive property enables multi-step demand forecasts where each future timestep conditions on prior predictions. Teacher forcing during training feeds ground-truth values instead of predictions to stabilize learning, while inference uses scheduled sampling to bridge the train-test discrepancy.

Attention solves the context vector bottleneck by allowing the decoder to dynamically access all encoder hidden states at each decoding step. Instead of relying on a single compressed vector, the decoder computes a weighted sum of encoder outputs, focusing on the most relevant historical periods. For demand forecasting, this means the model can attend to the same week last year when predicting seasonal peaks or recent days for short-term trends.

The encoder processes input sequences in both forward and backward directions using bidirectional RNNs or non-causal attention in Transformers. This allows each timestep's representation to incorporate both past and future context within the input window. For demand forecasting, a bidirectional encoder understands that a sales dip is followed by a promotional lift, creating richer representations than unidirectional processing.

Unlike fixed-window feedforward networks, encoder-decoder architectures naturally handle variable-length inputs and outputs through recurrent processing or positional encodings. This is critical for supply chain applications where historical data availability varies by SKU—new products have short histories while mature products have years of data. The architecture adapts without requiring padding, truncation, or separate models for different history lengths.

The fixed-length latent representation that forms the informational bridge between the encoder and decoder. The encoder compresses the entire input sequence into this dense vector, capturing its semantic essence. The decoder then conditions its generation on this vector. A key limitation of basic architectures is the information bottleneck: a single fixed-size vector struggles to retain all nuances of long input sequences, which motivated the development of attention mechanisms.

A computational module that allows the decoder to dynamically focus on different parts of the input sequence at each generation step, rather than relying solely on a static context vector. It computes a weighted sum of all encoder hidden states, where the weights reflect the relevance of each input token to the current decoding step. Self-attention and cross-attention are the core operations in the Transformer architecture, enabling parallel processing and capturing long-range dependencies.

A training strategy where the decoder receives the ground truth previous token as input, instead of its own prediction from the prior step. This stabilizes training by preventing the model from compounding early errors. During inference, this is replaced by autoregressive generation, where the model feeds its own output back as input. The discrepancy between teacher-forced training and autoregressive inference is known as exposure bias.

A heuristic search algorithm used during inference to find a high-probability output sequence. Instead of greedily selecting the single most likely token at each step, beam search maintains a fixed number (beam width) of candidate sequences. It expands all candidates, scores them, and retains the top-k. This balances exploration and exploitation, often yielding more globally coherent sequences than greedy decoding, at the cost of increased computational overhead.

The broader problem class and model family that encoder-decoder architectures are designed to solve. Seq2Seq models map an input sequence of arbitrary length to an output sequence of arbitrary length. Foundational applications include machine translation, text summarization, and time series forecasting. In supply chain contexts, a Seq2Seq model can ingest a sequence of historical demand signals and output a sequence of future demand predictions with quantified uncertainty.

The generation mode where the decoder produces output tokens one step at a time, conditioning each prediction on the previously generated tokens. This creates a causal structure where future tokens depend on past outputs. In probabilistic demand forecasting, an autoregressive decoder can generate a full trajectory of future demand samples, with each day's prediction conditioned on the previous day's sampled value, naturally capturing temporal dependencies.