Adaptive Moment Estimation (Adam)

Adam computes individual, adaptive learning rates for each parameter by maintaining an exponentially decaying average of past squared gradients ($v_t$). This mechanism effectively normalizes parameter updates, allowing weights associated with high-frequency features to receive smaller updates and weights for rare, high-cardinality features (like niche product IDs) to receive larger updates. This is critical for sparse categorical data common in CTR models, where a uniform global learning rate would cause instability.

Unlike pure adaptive methods like RMSprop, Adam incorporates momentum by maintaining an exponentially decaying average of past gradients ($m_t$). This acts as a velocity vector, smoothing the optimization trajectory and accelerating convergence in directions with consistent gradient sign. This helps the optimizer navigate the noisy, non-convex loss landscapes of deep factorization machines and multi-task architectures, dampening oscillations caused by mini-batch variance.

Standard exponential moving averages are initialized at zero, which biases the moment estimates toward zero during the initial training steps, especially when decay rates ($\beta_1, \beta_2$) are close to 1. Adam applies an explicit bias-correction step, dividing $m_t$ and $v_t$ by $(1 - \beta^t)$. This counteracts the initialization bias, ensuring that learning rates are not artificially suppressed at the start of training, leading to faster and more stable early convergence.

The element-wise normalization of the gradient by the square root of the second moment estimate makes Adam effectively scale-invariant. Rescaling the gradient vector by a constant factor (e.g., through a diagonal rescaling of the loss function) has no effect on the direction of the parameter update. This property makes the optimizer highly robust to the specific scale of the loss function and the variance of stochastic gradients, simplifying hyperparameter tuning across different model architectures like Deep Interest Networks.

While Adam is robust, its performance is sensitive to the step-size hyperparameter ($\alpha$). Standard practice involves a learning rate schedule, such as exponential decay or step-wise reduction, to refine convergence in the final phases of training. The exponential decay rates ($\beta_1$ for momentum, typically 0.9, and $\beta_2$ for adaptive scaling, typically 0.999) control the memory horizon of the moment estimates, effectively defining a window of past gradients that influence the current update direction.

CTR prediction often involves online learning and non-stationary data distributions due to shifting user behavior. Adam's per-parameter learning rates naturally decay as the second moment accumulator grows, which can be problematic for continuous learning. However, its momentum component allows it to track moving optima more effectively than purely adaptive methods, making it a strong baseline for online model retraining pipelines where the data distribution is constantly evolving.

Adam maintains an exponentially decaying average of past gradients (first moment, m_t) and past squared gradients (second moment, v_t). These act as estimates of the mean and uncentered variance. The first moment provides momentum, smoothing oscillations in ravines of the loss surface. The second moment scales the learning rate per-parameter, giving frequently occurring features smaller updates and rare features larger updates—critical for sparse categorical inputs like user IDs.

Since m_t and v_t are initialized to zero, they are biased toward zero during early training steps. Adam applies a bias correction factor that divides the moment estimates by (1 - β^t), where β is the decay rate and t is the timestep. This counteracts the initialization bias without requiring a separate warmup phase, ensuring stable and accurate updates from the very first mini-batch.

Standard hyperparameters for CTR models:

• Learning rate (α): Typically 0.001, often reduced on plateau
• β₁ (first moment decay): 0.9, controls momentum
• β₂ (second moment decay): 0.999, controls adaptation speed
• ε (epsilon): 1e-8, prevents division by zero These defaults from the original paper remain robust, though β₂ is sometimes reduced to 0.99 for non-stationary ad click distributions.

Vanilla SGD applies a uniform learning rate to all parameters, struggling with sparse gradients from one-hot encoded categorical features. Adam's per-parameter adaptive rates allow embeddings for rare item IDs to receive larger updates, rapidly learning their representations. This property makes Adam the default optimizer for models like DeepFM and DIN, where embedding tables dominate the parameter count.

Standard L2 regularization is not equivalent to weight decay in Adam due to the adaptive learning rate coupling. AdamW decouples weight decay from the gradient update, applying it directly to the weights: θ_t = θ_{t-1} - η * (α * m̂_t / (√v̂_t + ε) + λ * θ_{t-1}). This provides superior generalization in deep CTR models and is the recommended variant in modern frameworks like PyTorch and TensorFlow.

Adam's regret bound is O(√T) in the convex setting, matching the optimal rate. In non-convex deep CTR landscapes, empirical results show:

• Faster initial convergence than SGD with momentum
• Robustness to noisy gradients from mini-batch click data
• Reduced need for manual learning rate scheduling However, Adam can converge to sharper minima with slightly worse generalization than well-tuned SGD in some vision tasks—though this gap is negligible in CTR prediction.