Learning Rate Scheduler

A piecewise constant schedule where the learning rate is reduced by a multiplicative factor (e.g., 0.1) at predefined epoch intervals. This is a classic, interpretable schedule.

• Mechanism: After every N epochs, the learning rate is multiplied by a decay factor gamma.
• Typical Use: Foundational model fine-tuning where training stability is paramount.
• Hyperparameters: Initial learning rate, decay step (epoch interval), decay factor.

A schedule where the learning rate decays continuously by an exponential factor after each epoch or update step, providing a smooth decline.

• Formula: lr = initial_lr * (decay_rate ^ (step / decay_steps)).
• Behavior: Provides a smooth, aggressive early reduction that gradually plateaus.
• Application: Often used in scenarios where a rapid initial reduction in learning rate is beneficial to navigate sharp loss landscapes.

A schedule that decreases the learning rate from a maximum to a minimum following a half-cycle of a cosine curve, often combined with warm restarts (SGDR).

• Mechanism: lr = min_lr + 0.5 * (max_lr - min_lr) * (1 + cos(π * T_cur / T_i)).
• Key Benefit: The smooth, periodic reduction can help models escape local minima and converge to flatter, more generalizable solutions.
• SGDR Variant: Stochastic Gradient Descent with Warm Restarts periodically resets the learning rate to its maximum, simulating multiple training runs.

A composite schedule that linearly increases the learning rate from zero to a peak over a warmup period, then linearly decays it to zero over the remaining training steps.

• Warmup Phase: Mitigates instability during the initial phase of training, especially for large models and adaptive optimizers like Adam.
• Decay Phase: Provides a predictable, linear reduction to fine-tune weights. This is the default schedule in many transformer-based model training recipes (e.g., for BERT, GPT).
• Prevalence: The de facto standard for pre-training and fine-tuning large language models.

A aggressive schedule where the learning rate first increases from a low value to a high peak, then decreases back, all within a single training cycle, often combined with momentum cycling.

• Profile: Asymmetric, resembling a triangle. Learning rate peaks around the 30-50% point of training.
• Philosophy: Based on the idea of super-convergence, allowing for faster training with larger learning rates by carefully managing the schedule.
• Key Feature: Often uses a final annealing phase to a very low learning rate, helping the model settle into a minimum.

A dynamic, validation-driven schedule that reduces the learning rate by a factor when a monitored metric (e.g., validation loss) stops improving for a 'patience' number of epochs.

• Reactive, Not Proactive: The schedule adapts to the model's observed progress rather than following a fixed timeline.
• Key Hyperparameters: factor (e.g., 0.5), patience (epochs to wait), min_lr (lower bound).
• Use Case: Ideal for fine-tuning where the optimal number of epochs or decay steps is unknown, as it automatically responds to convergence signals.

For research or specialized needs, custom schedulers are often required.

PyTorch Pattern: Subclass torch.optim.lr_scheduler._LRScheduler and implement get_lr(self).

Example - Linear Warmup with Cosine Decay:

pythonCopy
class WarmupCosineSchedule(torch.optim.lr_scheduler._LRScheduler):
    def __init__(self, optimizer, warmup_steps, total_steps, last_epoch=-1):
        self.warmup_steps = warmup_steps
        self.total_steps = total_steps
        super().__init__(optimizer, last_epoch)
    def get_lr(self):
        if self.last_epoch < self.warmup_steps:
            # Linear warmup
            return [base_lr * self.last_epoch / self.warmup_steps for base_lr in self.base_lrs]
        else:
            # Cosine decay to zero
            progress = (self.last_epoch - self.warmup_steps) / (self.total_steps - self.warmup_steps)
            return [base_lr * 0.5 * (1 + math.cos(math.pi * progress)) for base_lr in self.base_lrs]

Key Considerations:

• Ensure get_lr returns a list of learning rates, one per parameter group.
• The base class handles the calling of get_lr and updating the optimizer.
• The last_epoch attribute tracks the number of step() calls.

Advanced training regimes may require chaining multiple scheduling policies (e.g., warmup followed by a plateau detector).

PyTorch's ChainedScheduler: Sequences multiple schedulers, transitioning from one to the next.

Common Pattern - Warmup + Plateau Reduction:

pythonCopy
# 1. Linear warmup for first 10 epochs
warmup_scheduler = torch.optim.lr_scheduler.LambdaLR(
    optimizer, lr_lambda=lambda epoch: epoch / 10)
# 2. Then monitor validation loss and reduce on plateau
plateau_scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(
    optimizer, mode='min', patience=5)

for epoch in range(epochs):
    train(...)
    val_loss = validate(...)
    if epoch < 10:
        warmup_scheduler.step()
    else:
        plateau_scheduler.step(val_loss)  # Note: requires metric

Important Distinction: ReduceLROnPlateau is an exception; it is a metric-based scheduler and its step() method requires a metric value as an argument. It cannot be easily chained with epoch/step-based schedulers using ChainedScheduler.

A critical implementation detail is when the scheduler's step() method is called, which differs by framework and scheduler type.

Epoch-Based: Called at the end of each training epoch.

• Default for most PyTorch schedulers (e.g., StepLR, MultiStepLR, CosineAnnealingLR).
• Used in classic, epoch-counted training loops.

Step-Based: Called after each optimizer update (batch).

• Default for Hugging Face Trainer and TensorFlow schedules.
• Required for schedules with warmup defined in steps (e.g., constant_with_warmup).
• More fine-grained control, aligning with total training steps.

Implementation Impact:

• For epoch-based, scheduler.step() is placed after the inner training loop.
• For step-based, scheduler.step() is placed inside the inner training loop, after optimizer.step().

PyTorch Conversion: Use scheduler = torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda) where lr_lambda is a function of the global step counter, not epoch.

A training paradigm that teaches a model to distinguish between similar (positive) and dissimilar (negative) data pairs. In retrieval, this is used to pull the embeddings of a query and a relevant document closer together in a shared vector space while pushing irrelevant documents apart. Key objectives include:

• Learning a semantically meaningful embedding space.
• Enabling efficient similarity search via cosine distance.
• Often implemented using losses like Triplet Loss or InfoNCE Loss.

A training strategy that identifies and uses challenging, semantically similar but irrelevant documents as negative examples. This improves a retriever's discrimination ability far beyond using random negatives. The process involves:

• Running an initial retrieval to find documents that are close to the query but not truly relevant.
• Using these 'hard negatives' during contrastive learning.
• This forces the model to learn fine-grained distinctions, significantly boosting precision.

A neural design where separate encoders independently map queries and documents into a shared embedding space. This architecture enables efficient approximate nearest neighbor search at inference time, as embeddings can be pre-computed and indexed. Characteristics include:

• Two independent transformer-based encoders (often sharing weights).
• Similarity is computed via dot product or cosine distance.
• The standard design for production-ready dense retrievers due to its speed.

The process of training a single, computationally intensive model that jointly processes a concatenated query and document to produce a relevance score. Unlike a dual-encoder, it performs deep, interactive attention across the pair. Primary use case:

• Reranking: Taking the top K results from a fast retriever (e.g., a dual-encoder) and re-scoring them for higher precision.
• It is too slow for first-stage retrieval over large corpora but provides superior accuracy for critical reordering.

A family of techniques that adapt large pre-trained models by training only a small subset of parameters. This is critical for fine-tuning retrievers without the prohibitive cost of full retraining. Common methods include:

• LoRA (Low-Rank Adaptation): Injects trainable low-rank matrices into model layers.
• Adapters: Inserts small, trainable modules between transformer layers.
• Enables rapid domain adaptation of retrievers with minimal GPU memory and storage overhead.

An advanced paradigm where the retriever and generator components of a RAG system are jointly optimized. This allows the retriever to learn based on the final generation quality, not just retrieval metrics. Key technical challenge:

• Backpropagation Through Retrieval: Passing gradients from the generator's loss back through the typically non-differentiable retrieval step, often solved with approximations like the straight-through estimator.
• Aims to create a more cohesive system where retrieval is directly informed by downstream utility.