Transformer

The core innovation of the Transformer. It allows each element in a sequence (e.g., a word) to directly attend to and aggregate information from all other elements, weighted by relevance. This creates dynamic, context-aware representations.

• Query, Key, Value Vectors: Each input is projected into these three vectors. The attention score between two positions is the dot product of the Query of one and the Key of the other.
• Scaled Dot-Product Attention: Scores are scaled by the square root of the key dimension to prevent vanishing gradients before applying a softmax to create a probability distribution.
• Parallel Computation: Unlike RNNs, all attention scores for a sequence can be computed simultaneously, leading to massive training speedups on parallel hardware like GPUs.

An extension where the self-attention mechanism is performed multiple times in parallel, each with different learned projection matrices. This allows the model to jointly attend to information from different representation subspaces.

• Heads: A standard Transformer might use 8 or 16 attention 'heads.'
• Diverse Focus: One head might learn to track syntactic dependencies (e.g., subject-verb agreement), while another tracks semantic relationships (e.g., coreference).
• Concatenated Outputs: The outputs from all heads are concatenated and linearly projected to form the final attention output, synthesizing the diverse information captured.

The core mathematical operation of the Transformer. It allows each token in a sequence to attend to all other tokens, computing a weighted sum of their values. The weights (attention scores) determine how much focus to place on each part of the input when encoding a specific position.

• Scaled Dot-Product Attention: The specific implementation used, which scales the dot product of queries and keys to prevent vanishing gradients.
• Enables parallelization: Unlike RNNs, all attention scores for a sequence can be computed simultaneously.
• Models long-range dependencies: Directly connects distant tokens, solving a key limitation of recurrent networks.

The original Transformer's two-stack design. The encoder processes the input sequence into a rich, contextualized representation. The decoder then uses this representation, along with the previously generated output, to produce the target sequence autoregressively.

• Encoder: A stack of identical layers, each with multi-head self-attention and a feed-forward network. Used for tasks like BERT (encoding only).
• Decoder: Similar layers, but with masked self-attention to prevent looking at future tokens during generation, and cross-attention to the encoder's output.
• Foundation for sequence-to-sequence tasks: Machine translation, text summarization, and conversational AI.

A method to inject order information into the model. Since the self-attention mechanism is permutation-invariant (it sees all tokens at once without inherent order), positional encodings are added to the input embeddings to give the model a sense of token position.

• Sinusoidal encodings: The original method uses sine and cosine functions of different frequencies, allowing the model to extrapolate to sequence lengths not seen during training.
• Learned embeddings: A common alternative where position embeddings are learned parameters, similar to token embeddings.
• Critical for understanding sequence structure: Enables the model to distinguish between 'dog bites man' and 'man bites dog'.

A Transformer-based generative model trained on vast text corpora via self-supervised learning (e.g., next-token prediction). LLMs leverage the Transformer's scalable architecture to capture complex linguistic patterns and world knowledge.

• Built on the Decoder stack: Models like GPT use a decoder-only Transformer architecture.
• Exhibits emergent abilities: Scaling laws show that capabilities like reasoning and in-context learning emerge with sufficient model size and data.
• Foundation for modern AI: The primary architecture behind ChatGPT, Claude, Llama, and other state-of-the-art generative AI systems.

The adaptation of the Transformer architecture to computer vision. An image is split into fixed-size patches, linearly embedded, and treated as a sequence of tokens fed to a standard Transformer encoder.

• Replaces Convolutional Neural Networks (CNNs): Achieves state-of-the-art results on image classification by modeling global relationships between patches from the first layer.
• Requires large-scale pre-training: Typically pre-trained on massive datasets like JFT-300M to compensate for lacking the inductive biases of CNNs.
• Spurred multi-modal models: The foundational approach for architectures that jointly process images and text (e.g., CLIP).

A parallelization and specialization technique within the self-attention mechanism. Instead of performing a single attention function, the model projects the queries, keys, and values into multiple, smaller subspaces and performs attention in parallel on each.

• Allows the model to jointly attend to information from different representation subspaces: One head might focus on syntactic relationships, another on coreference, etc.
• Increases representational capacity: The outputs of all heads are concatenated and linearly projected to form the final output.
• Standard in all Transformer blocks: A key hyperparameter is the number of attention heads (e.g., 8, 16, 32).