Feed-Forward Network (FFN)

Unlike the self-attention sub-layer, which mixes information across token positions, the FFN processes each token's representation in isolation. This means the exact same weights are applied to every vector in the sequence, treating each position identically. This design allows the attention mechanism to handle the routing of information between tokens, while the FFN focuses on refining what is stored within each token's hidden state.

The standard FFN consists of two linear projections separated by a non-linear activation function:

• Up-Projection: The first layer expands the input from the model's hidden dimension d_model to a larger intermediate dimension d_ff (often 4x larger).
• Activation: A non-linear function like ReLU, GELU, or SwiGLU is applied.
• Down-Projection: The second layer projects the expanded representation back down to d_model. This bottleneck structure forces the network to learn a compressed, expressive representation.

A leading interpretation views the FFN as a massive, static key-value memory bank. The first linear layer acts as a set of keys, detecting specific patterns in the input. The activation function selectively fires for matching patterns, and the second linear layer retrieves the corresponding values—factual knowledge or linguistic rules—and adds them to the residual stream. This is why factual recall is often localized to FFN weights.

Modern architectures like LLaMA and PaLM replace the simple ReLU with the SwiGLU activation function. This uses a gating mechanism:

• The input is projected into two separate matrices.
• One is passed through a Swish (SiLU) activation.
• The result is multiplied element-wise with the other matrix. This gated linear unit provides a smoother gradient flow and has been empirically shown to improve model quality, though it requires an additional weight matrix.

To scale model capacity without proportionally increasing compute, the single FFN can be replaced with a Mixture-of-Experts (MoE) layer. This contains multiple parallel FFN 'experts' and a trainable router network.

• Sparsity: For each token, the router selects only the top-k experts (e.g., 2 out of 8) to activate.
• Conditional Compute: This massively increases total parameters while keeping the FLOPs per token roughly constant, as inactive experts are not computed.

Despite its conceptual simplicity, the FFN dominates the parameter count and computational budget of a Transformer. In a standard architecture with d_ff = 4 * d_model, the two linear layers contain roughly two-thirds of the total model parameters. Optimizing these matrix multiplications is therefore the primary target for inference acceleration techniques like quantization and pruning.

The fundamental building block of the Transformer architecture. Each block consists of two primary sub-layers: a multi-head self-attention mechanism for mixing information between tokens, and a position-wise Feed-Forward Network for transforming each token's representation independently. Both sub-layers are wrapped with residual connections and layer normalization to stabilize training and enable gradient flow in very deep networks.

A pathway that adds the input of a sub-layer directly to its output before normalization. In the context of the FFN, the operation is LayerNorm(x + FFN(x)). This identity mapping prevents the vanishing gradient problem, ensuring that even if the FFN weights collapse to near-zero, the signal from the previous layer passes through unimpeded. This is the primary architectural enabler for training Transformers with hundreds of layers.

A normalization technique applied across the feature dimension of a single token's representation, rather than across the batch. It computes the mean and variance for each token independently, re-centering and re-scaling the activations. This reduces internal covariate shift during training and is typically applied before the FFN sub-layer in a Pre-LN architecture, which is the dominant configuration in modern models.

The activation function most commonly used within the FFN of modern Transformer models like BERT and GPT. Unlike ReLU, GELU applies a smooth, stochastic regularization by multiplying the input by the cumulative distribution function of the standard Gaussian. This provides a non-monotonic 'bump' near zero, allowing for more nuanced gradient flow compared to the hard zeroing of ReLU.

A gated activation variant used in models like LLaMA and PaLM to replace the standard two-layer FFN. It uses a SwiGLU (Swish-Gated Linear Unit) where the output is Swish(xW_1) ⊗ (xW_2). This introduces a multiplicative gating mechanism that dynamically controls information flow, often requiring a three-weight-matrix projection to maintain the same parameter count while improving training stability and downstream performance.

An architecture that replaces the single, dense FFN with multiple parallel 'expert' FFNs and a learned routing network. For each token, the router selects a sparse subset (e.g., top-2) of experts to activate. This dramatically increases total model parameters without proportionally increasing compute, as only a fraction of the FFN weights are used per token. Used in Mixtral and GPT-4.