Cost Function

The most prevalent cost function in DPD is the Mean Squared Error, defined as the expected value of the squared magnitude of the error signal—the difference between the desired linear output and the observed PA output.

• Formulation: J(w) = E[|y_desired(n) - y_observed(n)|²]
• Quadratic Nature: Creates a convex optimization surface with a single global minimum, guaranteeing convergence for gradient-based methods
• Gaussian Optimality: Statistically optimal when the distortion noise is additive white Gaussian
• Practical Implementation: Instantaneous squared error replaces the expectation for online Stochastic Gradient Descent

The quadratic penalty heavily weights large errors, aggressively suppressing spectral regrowth peaks that violate ACLR masks.

Pure MSE minimization often fails in practice due to ill-conditioning of the correlation matrix, especially with highly correlated basis functions or insufficient signal excitation.

• Tikhonov Regularization: Adds a penalty term λ||w||² to the cost function, where λ is the regularization parameter
• Effect on Solution: Biases coefficient estimates toward zero, trading a small increase in bias for a large reduction in variance
• Hardware Necessity: Essential for fixed-point FPGA implementations where matrix inversion is sensitive to finite-precision effects
• Levenberg-Marquardt: Extends regularization by adaptively adjusting λ based on the reduction in cost at each iteration

Without regularization, coefficient estimates can oscillate wildly or diverge entirely during background calibration.

Uniform weighting of all frequency components is often suboptimal. Frequency-weighted cost functions apply spectral shaping to prioritize linearization in critical bands.

• Spectral Mask Weighting: Assigns higher cost to errors in adjacent channels where ACLR regulatory limits are most stringent
• In-Band vs. Out-of-Band: Balances the trade-off between EVM (in-band signal quality) and spectral regrowth suppression
• Implementation: Achieved by filtering the error signal through a weighting filter before squaring, or by formulating the cost in the frequency domain
• Iterative Reweighting: Dynamically adjusts weights based on current distortion profile to focus adaptation where nonlinearity is most severe

This approach is critical for wideband signal linearization where PA memory effects vary across frequency.

The choice between sample-by-sample and block-based cost evaluation defines the adaptation architecture.

• Instantaneous Cost (LMS Family): J(w) = |e(n)|², updated every sample. Enables true closed-loop DPD with minimal latency but suffers from gradient noise
• Accumulated Cost (Block LS/RLS): J(w) = Σ|e(n)|² over a block of N samples. Provides more accurate gradient estimates at the cost of increased memory and processing delay
• Mini-Batch Compromise: Accumulates cost over small batches (e.g., 100-1000 samples) for Stochastic Gradient Descent, balancing gradient accuracy with update rate
• Forgetting Factor Integration: RLS uses λ (0.98-0.999) to exponentially weight recent errors, enabling tracking of thermal memory effects

The selection directly impacts convergence rate and the ability to track time-varying PA nonlinearity.

Unconstrained minimization may produce coefficients that violate hardware limitations. Constrained cost functions incorporate physical bounds directly into the optimization.

• Peak Power Constraint: Limits predistorter gain expansion to prevent driving the PA into deeper saturation or exceeding DAC full-scale
• Coefficient Magnitude Bounds: Restricts |w_k| ≤ w_max to prevent overflow in fixed-point FPGA-based DPD implementation
• Smoothness Regularization: Penalizes large differences between adjacent Look-Up Table entries to ensure monotonic predistorter response
• Lagrangian Methods: Incorporate constraints via Lagrange multipliers, solved using QR decomposition for numerical stability

Constrained formulations are essential when transitioning from floating-point simulation to resource-constrained hardware deployment.

Modern DPD systems often optimize multiple competing objectives simultaneously, requiring Pareto-optimal trade-off formulations.

• EVM + ACLR Joint Optimization: J(w) = α·EVM(w) + (1-α)·ACLR(w), where α balances in-band fidelity against out-of-band compliance
• Efficiency-Aware Cost: Incorporates PA drain efficiency into the cost, seeking coefficients that linearize while minimizing DC power consumption—critical for envelope tracking integration
• PAPR Penalty: Adds cost for signal peak regrowth caused by aggressive predistortion, protecting the PA from overdrive
• Multi-Band Cross-Terms: For multi-band DPD architectures, includes terms penalizing intermodulation products falling into other active bands

These formulations require careful weighting to avoid oscillating between objectives during online training algorithms.