Encoder-Decoder Architecture

The encoder is responsible for processing and compressing the variable-length input sequence (e.g., a sentence in French, an image grid) into a fixed-dimensional context vector or latent representation. This representation, often called the thought vector, aims to capture all the salient information needed for the decoder to perform its task. Common encoder architectures include:

• Recurrent Neural Networks (RNNs) like LSTMs or GRUs, which process tokens sequentially.
• Transformer Encoders, which use self-attention to process all input tokens in parallel, capturing long-range dependencies more effectively.
• Convolutional Neural Networks (CNNs) for encoding spatial data like images.

The decoder takes the context representation from the encoder and generates the output sequence (e.g., an English translation, an image caption) token-by-token. It is typically an autoregressive model, meaning it generates one token at a time, conditioning each new prediction on the encoder's context and its own previously generated tokens. Key components include:

• An initial state derived from the encoder's context.
• Autoregressive generation, where the output at step t becomes part of the input for step t+1.
• Attention mechanisms (especially in transformers) that allow the decoder to focus on different parts of the encoder's output for each generation step, overcoming the bottleneck of a single fixed context vector.

Introduced in the seminal paper 'Attention Is All You Need,' the attention mechanism revolutionized encoder-decoder designs. Instead of forcing the decoder to rely solely on a single, compressed context vector from the encoder, attention allows the decoder to dynamically attend to all of the encoder's hidden states at each generation step. This creates a direct, weighted connection between input and output tokens.

• Query, Key, Value: The decoder's current state is the query. It computes compatibility scores (attention weights) with all keys (derived from encoder states). These weights are used to create a contextualized summary from the values (also from encoder states).
• This enables handling of long input sequences, improves translation of rare words, and provides a form of interpretability via attention maps.

The Transformer is the dominant modern instantiation of the encoder-decoder pattern, eschewing recurrence entirely for stacked self-attention and feed-forward layers. Its pure attention-based design allows for massive parallelization during training.

• Encoder Stack: A series of identical layers, each with multi-head self-attention and a feed-forward network. It processes the input sequence to produce a set of contextualized representations.
• Decoder Stack: Similar layers, but with a masked multi-head self-attention layer (to prevent attending to future tokens) inserted before the cross-attention layer (where it attends to the encoder's output).
• This architecture is the backbone of models like BART, T5, and most modern sequence-to-sequence models.

Encoder-decoder architectures are the standard solution for a wide range of sequence-to-sequence and structured prediction problems:

• Machine Translation: The canonical use case (e.g., English → German).
• Text Summarization: Generating a concise summary from a long document.
• Image Captioning: The encoder is a CNN (e.g., ResNet) processing the image; the decoder is an RNN or transformer generating the caption.
• Speech Recognition: Encoder processes audio spectrograms; decoder generates transcribed text.
• Code Generation: Translating natural language descriptions or pseudocode into executable code.
• Conversational AI: Generating the next response in a dialogue given the conversation history.

The encoder-decoder concept has inspired several related designs:

• Encoder-Only Models (e.g., BERT): Use only the encoder stack for tasks like classification and named entity recognition, where the output is a label or tag for each input token.
• Decoder-Only Models (e.g., GPT series): Use only the decoder stack (with causal masking) for autoregressive generative tasks like text completion, where the model predicts the next token in a sequence.
• Multimodal Encoder-Decoders: Extend the paradigm to multiple input types. For example, a vision encoder (e.g., ViT) processes an image, and a language decoder (a transformer) generates a text description, forming the basis for Multimodal Large Language Models (MLLMs).

A neural network design for multimodal tasks where separate, independent encoders process each input modality (e.g., image and text). Their outputs are projected into a joint embedding space and aligned using a contrastive loss function like InfoNCE. This architecture is highly efficient for cross-modal retrieval tasks, as embeddings can be pre-computed and indexed. It is the core design behind models like CLIP and ALIGN.

• Key Feature: Enables fast, scalable retrieval via nearest-neighbor search in the shared space.
• Limitation: Lacks deep, token-level fusion between modalities during inference.

A neural network design where separate encoders first process each modality, followed by one or more fusion layers that deeply integrate the modalities. These layers often use cross-modal attention mechanisms, allowing tokens from one modality (e.g., text) to attend to features from another (e.g., image patches). This architecture produces a joint, task-specific representation and is commonly used for tasks requiring fine-grained reasoning, such as Visual Question Answering (VQA) and visual grounding.

• Key Feature: Enables complex, compositional reasoning across modalities.
• Trade-off: More computationally intensive than dual-encoder designs for inference.

A core neural network mechanism within transformer-based fusion architectures. It allows a sequence from one modality (the "query" modality) to attend to and integrate information from a sequence of another modality (the "key" and "value" modality). For example, in an image captioning model, each word token in the decoder can attend to all visual feature tokens from the encoder. This mechanism is fundamental for tasks that require fine-grained alignment, such as determining which image region the phrase "red car on the left" refers to.

An objective function used to train models like dual-encoders. InfoNCE loss treats learning as a noise-contrastive estimation problem: for a batch of image-text pairs, the model must identify the correct matching text (the positive) for a given image among all other texts in the batch (the negatives). It maximizes a lower bound on the mutual information between the modalities. This loss pulls the embeddings of matched pairs together in the joint space while pushing mismatched pairs apart.

• Mathematical Form: L = -log(exp(sim(I, T)/τ) / Σ exp(sim(I, T_j)/τ)) where sim is cosine similarity and τ is a temperature parameter.

A pre-training objective formulated as a binary classification task. The model, often a fusion-encoder, receives an image-text pair and must predict if they are correctly matched (positive) or randomly mismatched (negative). This requires the model to perform fine-grained cross-modal understanding beyond global alignment, as it must verify semantic consistency between specific details in the image and text. ITM is frequently used alongside contrastive loss (ITC) in comprehensive VLP frameworks like ALBEF to instill both global and local alignment capabilities.

A shared, high-dimensional vector space (e.g., 512 or 768 dimensions) where representations from different modalities are projected. The core principle is that semantically similar concepts from each modality (e.g., an image of a dog and the word "dog") are located near each other, as measured by cosine similarity or L2 distance. This space enables cross-modal retrieval, zero-shot classification, and serves as the foundational representation for many downstream vision-language tasks. Its quality is directly determined by the pre-training objectives (e.g., contrastive loss) used to learn it.