Transformer Block

The core sub-layer that allows the model to jointly attend to information from different representation subspaces at different positions.

• Mechanism: Projects the input into multiple sets of Query, Key, and Value (QKV) vectors, computes scaled dot-product attention in parallel, and concatenates the results.
• Purpose: Enables each token to dynamically weigh the importance of every other token in the sequence, capturing long-range dependencies.
• Example: In the sentence 'The cat sat on the mat because it was tired,' the head learns to link 'it' back to 'cat' with a high attention weight.

A fully connected network applied identically and independently to each token's representation after the attention sub-layer.

• Structure: Typically consists of two linear transformations with a non-linear activation function like GELU or ReLU in between.
• Function: Introduces non-linearity and processes the information gathered by the attention heads, expanding the hidden dimension to a larger intermediate size before projecting it back.
• Analogy: If attention is the 'communication' step, the FFN is the 'computation' step where each token processes what it has learned.

Pathways that add the original input of a sub-layer directly to its output, bypassing the transformation.

• Formula: Output = LayerNorm(Input + Sublayer(Input))
• Critical Role: Mitigates the vanishing gradient problem, enabling the successful training of very deep networks by allowing unimpeded gradient flow.
• Impact: Without residual connections, deep Transformers would be nearly impossible to train effectively, as the signal would degrade with depth.

A normalization technique applied across the feature dimension of a single token's representation, not across the batch.

• Operation: Computes the mean and variance of a token's embedding vector and normalizes it, followed by a learned affine transformation.
• Placement: The original 'Post-LN' places it after the residual addition, while modern 'Pre-LN' architectures apply it before the sub-layer for improved training stability.
• Benefit: Reduces internal covariate shift and accelerates convergence during training.

The inherent computational limitation of standard self-attention, where both memory and time complexity scale quadratically with the input sequence length.

• Complexity: O(n²) for an input sequence of length n, as every token attends to every other token.
• Consequence: Processing a 32k-token document requires 1,024 times more compute than a 1k-token document, making it prohibitive for very long contexts.
• Mitigations: Techniques like Sparse Attention, Sliding Window Attention, and FlashAttention are designed to overcome this bottleneck.

A critical architectural choice defining the order of operations within the Transformer block, significantly impacting training stability.

• Post-Norm (Original): x + Sublayer(LayerNorm(x)). Prone to gradient instability in deep models, requiring careful learning rate warm-up.
• Pre-Norm (Modern): x + Sublayer(LayerNorm(x)) is incorrect; the standard Pre-Norm is LayerNorm(x + Sublayer(x)) is also incorrect. The correct Pre-Norm formula is x + Sublayer(LayerNorm(x)). Actually, the standard Pre-Norm is LayerNorm(x + Sublayer(x)) is Post-Norm. Let's be precise: Pre-Norm applies normalization before the sub-layer: x + Sublayer(LayerNorm(x)). Post-Norm applies it after: LayerNorm(x + Sublayer(x)).
• Adoption: Pre-Norm is the de facto standard in modern models like GPT-3 and Llama because it allows for more stable training without extensive warm-up.