Encoder-Decoder Architecture

The architecture consists of two distinct stacks of Transformer blocks. The encoder processes the entire input sequence bidirectionally, creating a rich, context-aware representation for each token. The decoder then generates the output sequence step-by-step, attending to both its own previously generated tokens via causal self-attention and the encoder's output via cross-attention. This separation of concerns makes it ideal for sequence-to-sequence tasks like machine translation, where the input and output lengths differ.

The encoder compresses the entire input sequence into a fixed-size set of hidden state vectors. This final encoder output acts as a dense representation that must capture all relevant semantic and syntactic information from the source. The decoder's cross-attention mechanism then learns to query this representation, extracting the specific information needed at each generation step. This bottleneck forces the model to learn a robust, generalized internal representation of the input.

The critical link between encoder and decoder is cross-attention. In each decoder block, the Query vectors are derived from the decoder's current hidden state, while the Key and Value vectors come from the encoder's final output. This allows every token generation step to dynamically search the entire source sequence for relevant context. Unlike self-attention, cross-attention is not masked, giving the decoder full, bidirectional visibility into the input.

The decoder operates autoregressively, meaning it generates one token at a time, and each generated token is fed back as input for the next step. To prevent the decoder from peeking at future tokens during training, causal masking is applied to its self-attention layers. This ensures the model learns to predict the next token based only on past context, a process often referred to as teacher forcing during training.

A fundamental distinction is that the encoder uses bidirectional self-attention, allowing every token to attend to every other token in the input. This builds a deep, holistic understanding of the source. The decoder, however, uses unidirectional (causal) self-attention, preserving the left-to-right autoregressive property. This hybrid approach combines the deep comprehension of a bidirectional model with the generative capability of an autoregressive one.

The original Transformer, introduced in the paper 'Attention Is All You Need', was an encoder-decoder model designed for machine translation. The encoder reads the source language sentence (e.g., English), and the decoder generates the target language sentence (e.g., French). This paradigm extends to many tasks:

• Text Summarization: Long document in, concise summary out.
• Question Answering: Passage and question in, answer span out.
• Code Generation: Natural language description in, code snippet out.

The critical bridge between the encoder and decoder. In this mechanism, the Query vectors originate from the decoder's current hidden state, while the Key and Value vectors come from the encoder's final output. This allows every generated token in the target sequence to dynamically search for and extract relevant context from the entire input sequence, enabling precise alignment between source and target languages in translation tasks.

Applied strictly within the decoder's self-attention layers to preserve the autoregressive property. A mask matrix sets all values above the main diagonal to negative infinity before the softmax operation, ensuring that the prediction for token t can only depend on tokens at positions less than t. This prevents the model from cheating by looking at future tokens during training, forcing it to learn genuine next-token prediction.

Since the self-attention mechanism is inherently permutation-invariant, the model has no native sense of word order. Positional encodings are vectors added to the input embeddings to inject sequential information. In the original Transformer, this used fixed sinusoidal functions of different frequencies. Modern variants often use learned embeddings or Rotary Position Embeddings (RoPE) to better capture relative distances between tokens.

A stack of identical Transformer blocks that process the entire input sequence bidirectionally. Each block contains a multi-head self-attention layer and a position-wise feed-forward network. Because there is no causal mask, every token can attend to every other token simultaneously, building a rich, context-aware representation of the source sentence before the decoder ever begins its work.

Generates the output sequence one token at a time, autoregressively. Each decoder block contains three sub-layers: a masked self-attention layer, a cross-attention layer that queries the encoder output, and a feed-forward network. The output of the final decoder block is projected through a linear layer and a softmax to produce a probability distribution over the target vocabulary for the next token.

An inference optimization specific to the decoder. During autoregressive generation, recomputing the Key and Value vectors for all previously generated tokens at each step is wasteful. The KV Cache stores these vectors in memory, so only the new token's Query, Key, and Value need to be computed. This transforms the per-step complexity from quadratic to linear, dramatically accelerating text generation.