Least Mean Squares (LMS)

LMS operates as an instantaneous approximation of the steepest descent method. Rather than computing the true gradient of the mean squared error (MSE) cost surface—which requires ensemble averaging—LMS uses the squared instantaneous error as a noisy but unbiased estimate.

• Gradient Estimate: ∇J(n) ≈ -2e(n)x(n), where e(n) is the error signal and x(n) is the input vector
• Weight Update Rule: w(n+1) = w(n) + μe(n)x(n)
• Computational Cost: O(N) multiplications per iteration, where N is filter length
• No Matrix Inversion: Unlike RLS, LMS avoids O(N²) or O(N³) operations entirely

The algorithm converges in the mean toward the Wiener-Hopf solution, with the stochastic gradient noise causing a random walk around the optimal weight vector.

The step size parameter μ governs the tradeoff between convergence speed and steady-state accuracy. This single scalar controls the entire adaptation dynamics of the algorithm.

• Stability Bound: 0 < μ < 2/λ_max, where λ_max is the largest eigenvalue of the input autocorrelation matrix
• Practical Bound: 0 < μ < 2/tr(R), where tr(R) is the trace of the correlation matrix, approximated by N times the input power
• Large μ: Fast initial convergence but large misadjustment—the excess MSE above the Wiener minimum
• Small μ: Slow convergence but fine-grained steady-state performance with minimal coefficient jitter
• Eigenvalue Spread Sensitivity: Convergence slows when the input signal has high spectral dynamic range, as the algorithm must accommodate the slowest mode

The misadjustment M is approximately M ≈ μ·tr(R)/2, linearly proportional to step size and input power.

The Normalized LMS algorithm addresses LMS's sensitivity to input signal power fluctuations by normalizing the step size with the instantaneous input vector energy.

• Update Rule: w(n+1) = w(n) + [μ̃ / (||x(n)||² + δ)] e(n)x(n)
• Normalized Step Size μ̃: Dimensionless parameter typically in range 0 < μ̃ < 2
• Regularization δ: Small positive constant preventing division by zero during silent intervals
• Convergence Invariance: NLMS converges at a rate independent of input signal scaling, solving a normalized optimization problem at each iteration
• Stability Guarantee: Converges in the mean-square sense for 0 < μ̃ < 2 regardless of input statistics

NLMS is the de facto standard in digital predistortion applications where signal envelope power varies dramatically with modulation and traffic load.

LMS inherently tracks slowly time-varying systems without explicit forgetting mechanisms. The stochastic gradient noise that causes misadjustment in stationary environments becomes a beneficial dithering signal that enables parameter tracking.

• Tracking Lag: A systematic delay between estimated and true coefficients proportional to μ⁻¹ and the rate of system change
• Optimal Step Size Tradeoff: Faster tracking (larger μ) increases misadjustment; slower tracking (smaller μ) increases lag error
• Thermal Memory Effects: LMS naturally adapts to power amplifier thermal drift occurring over millisecond-to-second timescales
• Doherty Load Modulation: Tracks impedance variations in Doherty PAs as auxiliary amplifier bias conditions shift
• No Matrix Reset Required: Unlike RLS, LMS never suffers from covariance matrix windup or requires periodic reinitialization

The total error decomposes into estimation noise + tracking lag, with the optimal μ balancing these competing error sources.

Sign-based LMS variants drastically reduce hardware multiplier requirements by quantizing either the error signal, the input signal, or both to their sign bits.

• Sign-Error LMS: w(n+1) = w(n) + μ·sgn[e(n)]·x(n) — replaces error multiplication with a sign flip
• Sign-Data LMS: w(n+1) = w(n) + μ·e(n)·sgn[x(n)] — replaces input multiplication with sign selection
• Sign-Sign LMS: w(n+1) = w(n) + μ·sgn[e(n)]·sgn[x(n)] — reduces update to simple addition/subtraction, eliminating all multipliers
• Convergence Penalty: Sign-Sign LMS converges roughly 10-20× slower than standard LMS for equivalent steady-state error
• FPGA/ASIC Advantage: Eliminates DSP slices entirely, enabling ultra-compact implementations on resource-constrained fabrics

These variants are critical for massive MIMO DPD where hundreds of predistorter chains must operate in parallel within strict power budgets.

Leaky LMS introduces a coefficient decay term that prevents weight vector drift in poorly conditioned or low-persistence-of-excitation scenarios common in narrowband and multi-carrier DPD.

• Update Rule: w(n+1) = (1 - μα)w(n) + μe(n)x(n), where α is the leak factor
• Bounded Weights: Prevents unbounded growth when input correlation matrix is singular or nearly singular
• DC Offset Rejection: Naturally suppresses accumulation of DC bias in coefficient registers from finite-precision arithmetic
• Spectral Null Handling: Maintains stability when input signal has zero energy in certain frequency bands—a common occurrence in multi-carrier LTE/NR signals with unused resource blocks
• Bias Introduction: The leakage term introduces a small bias toward zero, trading unbiased estimation for numerical robustness

Leaky LMS is essential for fixed-point FPGA implementations where finite wordlength effects can otherwise cause coefficient overflow.

A direct variant of LMS that normalizes the step size by the power of the input signal vector. Key mechanism: The update term is divided by ||x(n)||² + δ, where δ is a small regularization constant. This prevents gradient noise amplification when input power fluctuates, making convergence more stable for non-stationary signals like speech or modulated waveforms. Trade-off: Slightly higher computational cost per iteration due to the norm calculation, but significantly improved robustness.

An adaptive algorithm that recursively updates the inverse of the input correlation matrix to achieve an order-of-magnitude faster convergence than LMS. Mechanism: Minimizes a weighted least squares cost function using the matrix inversion lemma. Critical parameters: The forgetting factor λ (typically 0.95–0.999) controls the memory of the algorithm. Trade-off: O(N²) computational complexity versus O(N) for LMS, making it unsuitable for long filter lengths in resource-constrained hardware.

The broader optimization framework from which LMS is derived. LMS is essentially SGD applied to the mean squared error cost function using a single-sample instantaneous gradient estimate. Key distinction: General SGD uses mini-batches; LMS uses a batch size of 1. Convergence behavior: The noisy gradient estimate causes LMS to wander around the minimum rather than converging exactly, producing a steady-state misadjustment proportional to the step size μ.

The theoretical foundation defining the optimal weight vector: w_opt = R⁻¹p, where R is the input autocorrelation matrix and p is the cross-correlation vector between input and desired signal. LMS iteratively approximates this solution without requiring matrix inversion. Significance: The Wiener solution represents the theoretical lower bound on mean squared error. The misadjustment of LMS measures how far the steady-state solution deviates from this optimum.

The fundamental trade-off in LMS design governed by the step size μ. Convergence rate: Larger μ produces faster initial convergence, with time constant τ ≈ 1/(2μλ_i) for each mode. Misadjustment: Steady-state excess error is proportional to μ · Tr[R]. Practical implication: A step size that converges in 100 samples may produce 10% misadjustment; reducing μ by a factor of 10 cuts misadjustment to 1% but slows convergence proportionally. This trade-off motivates variable step-size LMS variants.

The dominant architecture for applying LMS to digital predistortion. Mechanism: A copy of the predistorter is placed in a postdistorter configuration after the power amplifier. LMS minimizes the error between the postdistorter output and the predistorter input. Key advantage: Avoids the need for a PA inverse model and works with standard LMS updates. Assumption: Relies on the postdistorter-to-predistorter copy being valid, which holds when the PA characteristic is sufficiently smooth and memory effects are captured by the model structure.