Prediction Error Method (PEM)

Under Gaussian noise assumptions, PEM achieves the Cramér-Rao lower bound as the data record length approaches infinity. This means no other unbiased estimator can achieve lower variance. For finite samples, PEM provides the maximum likelihood estimate when the noise model is correctly specified, making it the theoretical benchmark against which all other estimation algorithms are compared.

PEM defines a cost function based on the one-step-ahead prediction error:

• ε(t,θ) = y(t) - ŷ(t|t-1; θ)
• The parameter vector θ is chosen to minimize a scalar norm of these errors
• Common choices include the quadratic norm V(θ) = (1/N) Σ ε²(t,θ) This formulation directly penalizes the model's inability to forecast future outputs from past data.

PEM operates on a unified predictor model that encompasses all standard linear black-box structures:

• ARX: Autoregressive with exogenous input
• ARMAX: ARX with moving average noise
• Output Error (OE): Noise-free dynamics with colored output noise
• Box-Jenkins (BJ): Independent parameterization of system and noise dynamics This generality allows PEM to handle arbitrary noise colorations without bias.

Unlike closed-form solutions such as Least Squares, PEM typically requires iterative numerical optimization (e.g., Gauss-Newton or Levenberg-Marquardt) because the predictor is a nonlinear function of the parameters for general model structures. This computational cost is the primary trade-off for PEM's statistical optimality. For Output Error models specifically, the gradient must be computed by filtering signals through the model's own sensitivity functions.

When the prediction errors are Gaussian and independent, minimizing the quadratic prediction error criterion is mathematically equivalent to maximizing the likelihood function. This equivalence provides a rigorous probabilistic foundation: PEM estimates are the parameters that make the observed data most probable under the assumed model structure. This connection extends to non-Gaussian cases through appropriate norm selection.

In Digital Pre-Distortion applications, PEM faces specific challenges:

• Real-time constraints often preclude iterative optimization, favoring recursive approximations like RPEM
• The high PAPR of communication signals can make the quadratic cost sensitive to outliers
• Model structure selection (memory depth, nonlinearity order) critically impacts both bias and variance
• PEM's theoretical efficiency assumes a stationary system, which power amplifiers with thermal memory may violate

The statistical parent of the Prediction Error Method. MLE selects model parameters that maximize the probability of observing the actual data. Under the assumption of Gaussian prediction errors, the PEM cost function (minimizing the sum of squared errors) is mathematically identical to the MLE solution.

• Asymptotic Efficiency: Both MLE and PEM achieve the Cramér-Rao lower bound as the sample size approaches infinity.
• Key Distinction: PEM is a more general framework that does not strictly require a probabilistic assumption to formulate the optimization problem.

A robust alternative to PEM when the model structure suffers from correlated residuals. If the prediction error is not white noise, PEM estimates become biased. The IV method uses an external 'instrument' signal—correlated with the input but uncorrelated with the disturbance—to extract unbiased parameter estimates.

• Use Case: Essential for closed-loop system identification where feedback contaminates the noise independence assumption.
• Optimal IV: A variant that achieves the same asymptotic accuracy as PEM when the noise model is correct.

The standard MATLAB function pem in the System Identification Toolbox is the de facto industry implementation. It numerically minimizes the prediction error cost function using iterative search algorithms like Gauss-Newton or Levenberg-Marquardt.

• Syntax: sys = pem(data, init_sys) refines an initial model using time or frequency-domain data.
• Output: Returns a state-space or polynomial model with estimated covariance matrices, enabling uncertainty quantification.

A model selection metric deeply tied to PEM theory. AIC provides a quantitative trade-off between model fit (prediction error variance) and model complexity (number of parameters). It is derived directly from the Kullback-Leibler divergence and the asymptotic properties of the maximum likelihood/PEM framework.

• Formula: AIC = log(V) + 2d/N, where V is the loss function, d is the number of parameters, and N is the sample size.
• Application: Used to select the optimal model order for ARX, ARMAX, or Box-Jenkins structures identified via PEM.

The iterative numerical engine that solves the PEM minimization problem. Since the prediction error is a nonlinear function of the parameters, a closed-form solution is impossible. Gauss-Newton uses a Jacobian-based approximation of the Hessian matrix to update parameters efficiently.

• Update Rule: θ_{k+1} = θ_k - (J^T J)^{-1} J^T ε
• Advantage: Exploits the sum-of-squares structure of the PEM cost function to avoid computing full second-order derivatives.

The most general linear polynomial model structure optimized by PEM. It explicitly separates the input-output dynamics (G) from the noise dynamics (H), providing maximum flexibility.

• Equation: y(t) = [B(q)/F(q)] u(t) + [C(q)/D(q)] e(t)
• PEM Advantage: Unlike simpler ARX models, PEM correctly handles colored noise disturbances by modeling them independently, preventing bias in the system transfer function estimate.