Stochastic Gradient Descent (SGD)

Unlike batch gradient descent, which computes the gradient over the entire dataset, SGD estimates the gradient using a single sample or a mini-batch of data at each iteration. This introduces noise into the update direction, which acts as a natural regularizer, helping the algorithm escape shallow local minima and saddle points in non-convex loss landscapes. The noisy gradient estimate is an unbiased estimator of the true gradient, meaning its expected value equals the batch gradient.

The learning rate (η) is the critical hyperparameter controlling the step size of each parameter update. It governs the trade-off between convergence rate and steady-state misadjustment:

• Too large: The algorithm diverges or oscillates wildly around the minimum.
• Too small: Convergence is impractically slow, and the optimizer may become trapped.
• Scheduling: Decaying the learning rate over time (e.g., inverse scaling or step decay) is a common strategy to ensure fast initial progress while guaranteeing fine-grained convergence to the optimum.

Pure stochastic updates (batch size = 1) are highly noisy and fail to exploit parallel hardware. Mini-batch SGD processes a small, randomly selected subset of samples (e.g., 32, 64, or 256) to compute a lower-variance gradient estimate. This approach maps efficiently to SIMD vector processors, GPUs, and FPGA-based accelerators. The batch size introduces a second hyperparameter that trades off gradient accuracy against computational throughput per iteration.

Momentum is a technique that accelerates SGD by accumulating a velocity vector of past gradients. The update becomes a weighted sum of the current gradient and the previous update direction, smoothing oscillations in high-curvature directions and amplifying consistent descent directions. The momentum coefficient (typically 0.9) acts as a friction term. This is formalized in algorithms like Nesterov Accelerated Gradient (NAG), which evaluates the gradient at a look-ahead position for improved correction.

Standard SGD applies a single global learning rate to all parameters, which is suboptimal for models with sparse features or varying gradient scales. Adaptive methods compute per-parameter learning rates based on historical gradient information:

• AdaGrad: Accumulates squared gradients, decaying the rate for frequently updated parameters.
• RMSProp: Uses an exponentially decaying average of squared gradients to prevent premature rate collapse.
• Adam: Combines momentum with RMSProp's adaptive scaling, incorporating bias correction for the initial time steps. Adam is the de facto default optimizer for deep learning.

A critical hyperparameter (often denoted η or α) that scales the gradient step size in SGD. In DPD, it directly governs the trade-off between convergence rate and steady-state misadjustment.

• High rate: Fast tracking of PA thermal changes but risks oscillation around the minimum EVM.
• Low rate: Smooth, stable coefficient estimates but may fail to track rapid Doppler or VSWR changes.
• Adaptive schedules: Often decayed over time or normalized by signal power (as in NLMS) to stabilize training on non-stationary waveforms.

A variant of LMS that normalizes the step size by the instantaneous power of the input basis vector. This makes convergence behavior independent of signal amplitude fluctuations.

• Update rule: w(n+1) = w(n) + (μ / (||x(n)||² + ε)) * e(n) * x(n)
• ε: A small regularization constant preventing division by zero during silent periods.
• DPD relevance: Essential for signals with high Peak-to-Average Power Ratio (PAPR), preventing coefficient divergence during power peaks.

The mathematical objective that SGD minimizes. In DPD, this is typically the Mean Squared Error (MSE) between the desired linear output and the observed PA output.

• Instantaneous cost: J(n) = |e(n)|² = |y_desired(n) - y_observed(n)|²
• Gradient computation: The derivative of J with respect to the predistorter coefficients provides the direction for the update.
• Alternative costs: Some architectures use weighted error functions to prioritize ACLR reduction over in-band EVM.

The speed at which SGD drives the error signal to a steady-state floor. It is measured in iterations or time required to reach a specified EVM target.

• Bottleneck: Dominated by the eigenvalue spread of the correlation matrix of the basis functions.
• Misadjustment: The excess steady-state error above the theoretical Wiener optimum, caused by gradient noise in stochastic updates.
• Trade-off: Faster convergence inherently increases gradient noise, raising the misadjustment floor. This is the fundamental tension in online DPD tuning.