Masked Language Modeling (MLM)

BERT, introduced by Google in 2018, was the first model to popularize MLM as a core pre-training objective. Its key innovation was bidirectional context learning: by masking 15% of input tokens and training the model to predict them, BERT could consider both left and right context simultaneously. This was a departure from previous left-to-right or right-to-left language models.

• Architecture: Stack of Transformer encoder layers.
• Variants: BERT-Base (110M parameters), BERT-Large (340M parameters).
• Impact: Achieved state-of-the-art results on 11 NLP benchmarks at release, including GLUE, SQuAD, and SWAG, establishing a new paradigm for language model pre-training.

RoBERTa, from Meta AI, is not a new architecture but a replication study and hyperparameter optimization of BERT's training procedure. It demonstrated that BERT was significantly undertrained. Key modifications included:

• Removing the Next Sentence Prediction (NSP) objective, finding MLM alone was sufficient.
• Training with much larger batches (8K vs. 256) and more data (160GB vs. 16GB).
• Training for longer over more diverse data.
• Dynamically changing the masked pattern for each training sequence.

These changes led to substantial improvements over the original BERT, solidifying best practices for MLM-based pre-training.

ALBERT addressed the parameter inefficiency and memory constraints of large BERT models through two primary innovations while retaining the MLM objective.

• Factorized Embedding Parameterization: Separated the size of the hidden layers from the size of the vocabulary embeddings, drastically reducing parameters when the hidden size was increased.
• Cross-Layer Parameter Sharing: Shared parameters across all Transformer encoder layers, which significantly reduced the total parameter count.
• Inter-sentence coherence loss: Replaced BERT's NSP with a sentence-order prediction (SOP) loss for better discourse understanding.

This allowed ALBERT to achieve competitive performance with models having 18x fewer parameters, enabling more scalable deployments.

DeBERTa enhanced the standard MLM framework by improving how the model represents content and position. Its core innovations are:

• Disentangled Attention: Each word is represented using two separate vectors for its content and its relative position. Attention scores are computed as a sum of content-to-content, content-to-position, and position-to-content terms, allowing for more precise modeling of dependencies.
• Enhanced Mask Decoder (EMD): Replaced the output softmax layer in the MLM head with an absolute position-aware layer during pre-training, improving the model's use of positional context for masked token prediction.

DeBERTa outperformed RoBERTa on the SuperGLUE benchmark, surpassing human baseline performance for the first time.

ELECTRA introduced a sample-efficient pre-training task as an alternative to MLM. Instead of predicting masked tokens, it trains a discriminator to classify whether each token in the input is an original or a plausible replacement generated by a small generator network (trained with MLM).

• Mechanism: The generator corrupts the input (e.g., via MLM), and the discriminator (the main model) is trained to identify the replaced tokens.
• Advantage over MLM: It learns from all input tokens in every training step, not just the 15% that are masked, leading to greater computational efficiency. ELECTRA achieved GLUE scores comparable to RoBERTa using less than 25% of the compute.

SpanBERT extended the MLM objective by masking and predicting contiguous random spans of text rather than individual tokens.

• Span Masking: Randomly masks contiguous spans of tokens, with span lengths drawn from a geometric distribution.
• Span Boundary Objective (SBO): In addition to predicting the masked span's content, the model uses the representations of the tokens at the boundaries of the masked span to predict each token within it. This encourages the model to better leverage the surrounding context.

This approach proved particularly effective for tasks requiring span-level understanding, such as question answering (e.g., SQuAD) and coreference resolution, where it outperformed BERT.

Masked Image Modeling (MIM) is the direct visual analogue to MLM. Instead of text tokens, a model must reconstruct randomly masked patches of an image. This self-supervised objective teaches models to learn robust, general-purpose visual representations by predicting the content of missing regions based on surrounding context.

• Core Mechanism: Images are split into patches, a large random subset (e.g., 75%) is masked, and the model predicts the missing visual content, often as raw pixels or discrete visual tokens.
• Key Benefit: Enables pre-training of vision transformers (ViTs) without reliance on manually labeled classification datasets like ImageNet.
• Example: Models like BEiT and MAE use MIM to achieve state-of-the-art performance on downstream tasks like image classification and segmentation.

Contrastive Learning is a self-supervised paradigm that teaches a model to distinguish between similar and dissimilar data points, often used in tandem with MLM. While MLM is a generative objective (predicting missing content), contrastive learning is a discriminative objective (learning similarity).

• Core Mechanism: The model learns to pull representations of positive pairs (e.g., different augmentations of the same image, or a matched image-text pair) closer in an embedding space while pushing negative pairs apart.
• Common Loss Functions: InfoNCE Loss is the standard objective, derived from maximizing mutual information between positive pairs.
• Multimodal Use: Foundational models like CLIP are trained purely with a contrastive objective on image-text pairs, aligning the two modalities in a shared embedding space.

Causal Language Modeling (CLM), also known as autoregressive modeling, is the primary pre-training objective for decoder-only models like GPT. It predicts the next token in a sequence given all previous tokens, learning a unidirectional context.

• Key Difference from MLM: MLM is bidirectional—it uses context from both left and right to predict a masked token. CLM is strictly left-to-right, using only prior context.
• Architectural Implication: CLM requires a masked self-attention mechanism that prevents tokens from attending to future tokens, making it inherently sequential.
• Primary Use: Excellent for text generation tasks. Many modern architectures (e.g., LLaMA, GPT) combine MLM-style pre-training for representation learning with CLM fine-tuning for instruction following and chat.

Permutation Language Modeling is an objective that generalizes both MLM and CLM by training a model to predict tokens in a random, autoregressive order. It was introduced in the XLNet architecture to overcome the pretrain-finetune discrepancy of MLM.

• Core Idea: Instead of masking tokens, the model is trained to predict each token in a sequence given a random permutation of the other tokens. This allows it to learn from all possible factorization orders, capturing bidirectional context without artificial [MASK] tokens.
• Advantage over MLM: Eliminates the discrepancy where [MASK] tokens appear during pre-training but not during fine-tuning for tasks like text generation or question answering.
• Mechanism: Uses two-stream self-attention to maintain the identity of the token being predicted while preventing information leakage from its future content.

Span Corruption, or Masked Span Prediction, is a variant of MLM where contiguous spans (multiple consecutive tokens) are masked instead of individual tokens. This is the core pre-training objective for encoder-decoder models like T5.

• Core Mechanism: Random contiguous segments of text are replaced with a single sentinel token (e.g., <extra_id_0>). The decoder is then trained to reconstruct the original span of text autoregressively, conditioned on the corrupted encoder input.
• Key Benefit: It frames nearly all NLP tasks as a unified text-to-text problem, where the model learns to "translate" an input text (e.g., a question) into an output text (e.g., an answer).
• Efficiency: Predicting spans is more computationally efficient than predicting many individual tokens separately and encourages learning longer-range dependencies.

Replaced Token Detection (RTD) is a discriminative alternative to MLM, where the model must identify which tokens in a corrupted sequence have been replaced by a generator network. It was introduced in the ELECTRA model.

• Two-Model System: A small generator (often an MLM model) corrupts the input by replacing some tokens. A larger discriminator (the main model) is then trained to classify every token as "original" or "replaced."
• Efficiency Advantage: Unlike MLM, which learns only from the 15% of masked tokens, RTD provides a learning signal from every token in the sequence, leading to better sample efficiency.
• Outcome: ELECTRA demonstrated superior performance to BERT (trained with MLM) using the same compute budget, establishing RTD as a powerful pre-training paradigm.