Masked Language Modeling (MLM)

Unlike autoregressive models that predict tokens using only leftward context, MLM allows the model to attend to tokens on both sides of a masked position. This enables a deeper understanding of word meaning from its full surrounding context, which is critical for tasks like sentence classification and named entity recognition.

• Mechanism: The model sees the entire input sequence, with some tokens replaced by a special [MASK] token.
• Benefit: Captures dependencies in both directions, leading to richer contextual embeddings.
• Example: For the masked sentence "The capital of France is [MASK].", the model uses "France" and "capital" to predict "Paris".

A stochastic process determines which tokens are masked in each training iteration, preventing the model from simply memorizing patterns.

• Standard Rate: In BERT, 15% of tokens in the input sequence are selected for masking.
• Procedure: Of the selected 15%:
  • 80% are replaced with the [MASK] token.
  • 10% are replaced with a random token from the vocabulary.
  • 10% are left unchanged.
• Purpose: The random replacement and unchanged tokens force the model to develop a robust, distributional understanding of each word, not just a pattern for the mask token itself.

MLM is framed as a classification problem at the vocabulary level for each masked position. The model's output is a probability distribution over its entire vocabulary.

• Architecture: A linear classification layer (often called the MLM head) sits on top of the final hidden state corresponding to each [MASK] token.
• Objective: The model is trained to maximize the probability of the original, correct token. This is typically done using a cross-entropy loss.
• Scale: For a model with a 30,000-token vocabulary, it performs 30,000-way classification for each masked position. This dense prediction task is computationally intensive but teaches fine-grained linguistic knowledge.

MLM generates its own training labels from unlabeled text corpora, eliminating the need for costly human annotation. The input text serves as both the question and the answer.

• Pretext Task: The act of predicting masked tokens is an artificial, self-created task designed to teach the model general language structure.
• Data Efficiency: Enables learning from massive datasets like Wikipedia and BookCorpus, which would be infeasible to label manually.
• Transfer Learning: The rich, general-purpose representations learned via MLM can then be fine-tuned with a small amount of labeled data for specific downstream tasks (e.g., sentiment analysis, question answering).

MLM is intrinsically linked to the Transformer encoder architecture. The encoder's ability to process all tokens in parallel and compute attention between all pairs is essential for the bidirectional nature of MLM.

• Core Component: The multi-head self-attention mechanism allows any token (including masks) to attend to and aggregate information from all other tokens in the sequence.
• Contrast with Decoders: Autoregressive decoder-only models (like GPT) use causal masking to prevent looking ahead, making them unsuitable for the standard MLM objective.
• Variants: The success of MLM in BERT led to its adaptation in vision (Masked Image Modeling) and multimodal models, where patches of an image or tokens from another modality are masked and predicted.

While powerful, MLM has inherent constraints that influence model design and deployment.

• Pre-training/Fine-tuning Discrepancy: The model never sees [MASK] tokens during fine-tuning on downstream tasks, creating a train-test mismatch. The random replacement strategy mitigates this.
• Computational Cost: Predicting a small percentage of tokens requires a full forward pass for the entire sequence, which is less efficient than autoregressive next-token prediction for text generation.
• Not for Generation: MLM-trained encoders are not designed for open-ended sequence generation. They are typically used for understanding tasks or paired with a separate decoder for generation (e.g., in T5 or encoder-decoder models).

Masked Image Modeling (MIM) is the direct visual analogue to MLM. In this self-supervised pre-training objective, a random portion of an input image (e.g., patches or pixels) is masked or corrupted. The model, typically a Vision Transformer (ViT), is trained to reconstruct the original visual content of the masked regions based on the surrounding visible context.

• Mechanism: Common corruption methods include replacing patches with a learnable [MASK] token, random noise, or using an autoencoder to tokenize the image into discrete visual tokens for prediction.
• Purpose: Forces the model to learn rich, holistic representations of visual structure, object parts, and scene semantics, rather than just superficial features.
• Examples: BEiT, MAE, and SimMIM are prominent models using MIM. MAE, for instance, masks up to 75% of image patches and uses an asymmetric encoder-decoder to perform reconstruction.

Contrastive Language-Image Pre-training (CLIP) employs a fundamentally different objective than MLM. Instead of generative reconstruction, it uses contrastive learning to align representations from two different modalities.

• Architecture: Uses a dual-encoder architecture with separate image and text encoders.
• Objective: The model is trained on vast datasets of image-text pairs. The InfoNCE loss is used to maximize the similarity between the embeddings of matched (positive) pairs and minimize the similarity for all other (negative) in-batch combinations.
• Outcome: Creates a joint embedding space where semantically similar images and texts are close, enabling powerful zero-shot transfer for tasks like image classification and cross-modal retrieval without task-specific fine-tuning.

Self-supervised learning (SSL) is the overarching paradigm that includes MLM. In SSL, the model generates its own supervisory signal from the inherent structure of unlabeled data by solving a pretext task.

• Core Idea: Leverages the data itself as the source of truth. For text, the pretext task is predicting masked words (MLM). For images, it could be solving jigsaw puzzles or, as in MIM, reconstructing masked patches.
• Advantage: Enables learning from the vast quantities of unlabeled data available on the internet, which is far more scalable than relying solely on expensive human-annotated datasets.
• Broader Context: MLM for language and MIM for vision are specific instantiations of SSL. Other SSL methods include contrastive learning (e.g., SimCLR, CLIP) and autoregressive prediction (e.g., GPT).

The encoder-decoder architecture is a foundational neural design pattern used to implement MLM and related sequence-to-sequence tasks.

• Encoder Role: Processes the entire input sequence—including masked tokens—into a contextualized representation. In BERT's MLM, only the encoder is used, and a small classification head predicts the masked tokens.
• Decoder Role: In full sequence-to-sequence models (e.g., T5, BART), a decoder autoregressively generates an output sequence from the encoder's representation. For MLM-style tasks in these models, the output is the sequence of masked tokens.
• Contrast with BERT: BERT uses a encoder-only architecture for MLM, where the prediction is made in a non-autoregressive, parallel manner. This bidirectional context is key to its success but differs from the autoregressive generation of decoder-only models like GPT.

Pre-training objectives are the specific tasks used to train foundation models on large-scale data. MLM is one of several critical objectives used to instill general-purpose knowledge.

Key objectives in vision-language pre-training include:

• Masked Language Modeling (MLM): Predict masked text tokens.
• Masked Image Modeling (MIM): Predict masked image regions.
• Image-Text Contrastive (ITC): Align global image and text embeddings (used in CLIP).
• Image-Text Matching (ITM): A binary classification task to predict if an image and text pair are correctly matched, requiring fine-grained understanding.

Modern models like ALBEF or BLIP often use a multi-objective loss, combining MLM, ITC, and ITM during pre-training to learn both aligned representations and generative capabilities.

Cross-modal attention is the neural mechanism that enables deep integration between modalities, a step beyond the separate processing seen in dual-encoder models like CLIP.

• Function: Allows tokens from one modality (e.g., words) to directly attend to, and incorporate information from, tokens in another modality (e.g., image patches). It is typically implemented as a cross-attention layer within a transformer.
• Contrast with MLM: Standard MLM in BERT uses self-attention (text-to-text). Cross-modal attention is essential for fusion-encoder architectures used in tasks requiring deep reasoning, such as Visual Question Answering (VQA) or detailed image captioning.
• Use Case: In a model pre-trained with MLM on text and MIM on images, cross-modal attention layers would be added later to fuse these separately learned representations for joint multimodal tasks.