Transformer Encoder

Applied independently and identically to each position after attention, this sub-layer provides non-linear transformation and dimensionality expansion. It typically consists of two linear transformations with a ReLU activation in between.

• Position-Independent Processing: The same network is applied to every token's representation, but because each token's representation has already been contextualized by attention, the output is unique.
• Dimensionality Expansion: The hidden layer often expands to a higher dimension (e.g., 2048) before projecting back down, allowing the model to learn more complex features.
• Role in the Layer: While self-attention mixes information across positions, the FFN processes and refines the information at each individual position.

These are critical stabilization techniques that enable the training of very deep encoder stacks. Each sub-layer (attention, FFN) is wrapped with a residual connection followed by layer normalization.

• Residual Connection: The input to a sub-layer is added to its output (x + Sublayer(x)). This creates a direct pathway for gradients, mitigating the vanishing gradient problem and allowing information to flow through many layers.
• Layer Normalization: Applied to the sum of the residual, it stabilizes the activations by normalizing across the feature dimension for each sample independently. This leads to faster convergence and more stable training.
• Pre-LN vs. Post-LN: Modern architectures often use Pre-LayerNorm, where normalization is applied before the sub-layer, improving training stability for very deep models.

Since the self-attention mechanism is inherently permutation-invariant, positional encodings are added to the input embeddings to inject information about the order of tokens in the sequence. The original Transformer uses fixed, sinusoidal functions.

• Sinusoidal Encodings: Use sine and cosine functions of different frequencies, allowing the model to generalize to sequence lengths longer than those seen during training.
• Learnable Positional Embeddings: An alternative where position embeddings are learned parameters, common in models like BERT. These are effective but may not generalize as well to unseen lengths.
• Relative Position Representations: More advanced methods encode the relative distance between tokens (e.g., "key i is 3 positions before query j"), which can improve performance on tasks requiring precise positional reasoning.

The full encoder is composed of N identical layers (often 6, 12, or 24) stacked on top of each other. Each layer refines the representation passed from the layer below.

• Progressive Abstraction: Lower layers capture local syntax and phrase-level patterns, while higher layers build more abstract, semantic representations of the entire sequence.
• Feature Reuse: The residual connections facilitate feature reuse across layers, preventing degradation as depth increases.
• Model Scaling: Increasing the number of encoder layers (N), the model dimensionality (d_model), and the number of attention heads are the primary axes for scaling model capacity, as defined in the scaling laws for Transformers.

The final output of the encoder is a sequence of vectors, one for each input token, where each vector is a contextualized representation informed by the entire input sequence. These are used directly by decoder layers in sequence-to-sequence models or as features for downstream tasks.

• Sequence-Level Context: The representation for the word "bank" will differ if the context is "river bank" versus "investment bank," as determined by the global attention mechanism.
• Foundation for BERT: Models like BERT use only the Transformer encoder. The final hidden state of the special [CLS] token is often used as an aggregate sequence representation for classification.
• Feature Extraction: The output representations from intermediate encoder layers can be used as features for other models, a technique common in transfer learning.

The Attention Mechanism is the core computational unit within a Transformer encoder. It allows the model to dynamically weigh the importance of all other tokens in a sequence when encoding a specific token. This creates contextualized representations where the meaning of a word (or a visual patch) is informed by its surrounding context.

• Self-Attention: The specific form used in encoders, where the input sequence attends to itself.
• Query, Key, Value: The three learned linear projections used to compute attention scores and output weighted sums.
• Enables the model to handle long-range dependencies, a key advantage over recurrent neural networks.

Multi-Head Attention is an extension of the basic attention mechanism where the self-attention operation is performed in parallel multiple times (the "heads"). Each head learns to focus on different types of relationships or aspects of the input data.

• The outputs of all heads are concatenated and linearly projected to form the final output.
• For example, in a sentence, one head might attend to syntactic dependencies while another focuses on semantic coreference.
• This parallel design increases the model's representational capacity and its ability to capture diverse patterns simultaneously.

Positional Encoding is a method for injecting information about the order of tokens into a Transformer model. Since the self-attention mechanism is inherently permutation-invariant, positional encodings are added to the input embeddings to give the model a sense of sequence position.

• Absolute Positional Encoding: Uses fixed, sinusoidal functions of different frequencies to encode position indices.
• Relative Positional Encoding: Encodes the distance between tokens, which can generalize better to longer sequences than seen during training.
• Critical for tasks where order matters, such as language understanding or temporal sensor data processing.

The Feed-Forward Network is a component within each Transformer encoder layer that operates independently and identically on each token position. It follows the multi-head attention sub-layer and is responsible for applying a non-linear transformation to the attended representations.

• Typically consists of two linear layers with a ReLU or GELU activation in between.
• It can be viewed as a position-wise perceptron that further processes the contextual information gathered by attention.
• This adds model capacity and is a key part of the layer's ability to compute complex feature interactions.

A Vision Transformer is a model architecture that adapts the standard Transformer encoder, originally designed for sequences, to process images. It treats an image as a sequence of fixed-size non-overlapping patches, which are linearly embedded and fed into a standard Transformer encoder stack.

• Patch Embedding: The process of splitting an image into patches and projecting them into a vector space.
• A class token is often prepended to the sequence, whose final state is used for image classification.
• ViTs demonstrated that pure Transformer architectures, without convolutional inductive biases, can achieve state-of-the-art results on major image recognition benchmarks.

The Encoder-Decoder Architecture is a framework where a Transformer encoder processes an input sequence into a set of contextual representations, which a Transformer decoder then uses to autoregressively generate an output sequence. This is the original architecture used for sequence-to-sequence tasks like machine translation.

• The encoder's role is to comprehensively understand the input.
• The decoder uses cross-attention to attend to the encoder's output while generating tokens.
• While our focus is on the encoder, understanding its role in this larger system is fundamental. In VLA models, the encoder often processes visual and language inputs, while a decoder may generate action tokens.