Cumulant-Based Whitening

Whitening applies a linear transformation to the observed data such that its covariance matrix becomes the identity matrix. This forces all channels to have unit variance and zero cross-correlation, removing the spatial color introduced by antenna geometry, propagation, and receiver front-ends. The whitening matrix is derived from the inverse square root of the sample covariance matrix, typically computed via eigenvalue decomposition or Cholesky factorization.

A critical property of cumulant-based whitening is that it does not alter the higher-order statistics of the source signals. While second-order correlations are eliminated, the fourth-order cumulants, kurtosis, and skewness that encode modulation-specific signatures remain intact. This orthogonality ensures that whitening is a lossless preprocessing step for cumulant-based classifiers, preserving the non-Gaussianity that distinguishes QAM from PSK constellations.

When additive noise is spatially correlated across array elements—common in compact antenna arrays with mutual coupling—cumulant-based whitening simultaneously decorrelates the noise component. This transforms colored noise into spatially white noise, simplifying the statistical model for downstream algorithms. The process effectively equalizes the noise floor across channels, improving the performance of eigenvalue-based source enumeration and subspace methods.

Whitening can serve as an implicit dimensionality reduction step. By analyzing the eigenvalue spectrum of the covariance matrix, the whitening transform can project data onto only the signal subspace, discarding noise-only dimensions. This reduces the effective number of channels from the physical array count to the estimated number of sources, lowering the computational complexity of subsequent Joint Approximate Diagonalization (JADE) or Independent Component Analysis (ICA) algorithms.

Cumulant-based whitening normalizes the power of each output channel to unity. This eliminates arbitrary amplitude scaling factors that arise from unknown path loss and receiver gain variations. For modulation classification, this scale invariance is essential—it ensures that normalized cumulant features such as |C40|/|C42|² are computed on properly scaled data, preventing amplitude mismatches from corrupting the decision statistic.

The whitening transform requires estimating and inverting the sample covariance matrix, an O(N³) operation for N channels. For real-time streaming applications, recursive whitening algorithms update the whitening matrix incrementally with each new sample using rank-1 updates to the eigenvalue decomposition. Hardware implementations on FPGAs often use QR decomposition-based approaches that avoid explicit matrix inversion, achieving microsecond-latency whitening for tactical signal processing.

Whitening transforms the received signal vector z = Wx such that the covariance matrix becomes the identity matrix E[zz^H] = I. This forces all channels to have unit variance and zero cross-correlation, simplifying the subsequent search for independent components to a unitary rotation problem. In a 4-element uniform linear array receiving mixed QPSK and 16QAM signals, whitening removes the spatial color introduced by the array geometry and propagation environment, reducing the degrees of freedom the ICA algorithm must explore.

The Joint Approximate Diagonalization of Eigenmatrices (JADE) algorithm requires whitened data as a mandatory preprocessing step. After whitening, the fourth-order cumulant tensor is computed on the spatially-white data, and a set of eigenmatrices is extracted. JADE then finds a unitary matrix U that jointly diagonalizes these matrices. The complete separating matrix becomes B = UW, where W is the whitening matrix. This two-stage approach—whiten then rotate—is the standard architecture for cumulant-based blind source separation of communication signals.

When the additive noise is spatially white (uncorrelated across channels), whitening normalizes the noise floor to unity across all dimensions. This is critical because higher-order cumulants are theoretically blind to Gaussian noise only when the noise covariance structure is properly handled. By whitening the received data, the signal subspace and noise subspace are balanced, preventing the cumulant estimation from being dominated by channels with disproportionately high noise power. This improves the SNR wall for subsequent modulation classification.

Principal Component Analysis (PCA) whitening projects the data onto the signal subspace by retaining only eigenvectors corresponding to eigenvalues above the noise floor. For a scenario with 8 array elements but only 3 active co-channel signals, PCA whitening reduces the data dimension from 8 to 3. The transformation is W = D^{-1/2} E^H, where D contains the top eigenvalues and E the corresponding eigenvectors. This not only whitens but also performs source enumeration, telling the system how many signals to separate before cumulant processing begins.

In small-sample regimes common in burst-mode communications, the sample covariance matrix is ill-conditioned. Shrinkage whitening regularizes the estimate by shrinking the sample covariance toward a structured target (e.g., the identity matrix): Σ_shrunk = (1-ρ)Σ_sample + ρI. The shrinkage intensity ρ is chosen to minimize the Frobenius risk. This prevents the whitening matrix from amplifying estimation noise in small eigenvalues, which would otherwise corrupt the higher-order cumulant estimates used for modulation identification.

For real-time spectrum monitoring, batch whitening is replaced with recursive whitening using the matrix inversion lemma. The whitening matrix is updated with each new sample vector x(t) without storing the entire data history. The recursive update maintains a running estimate of the covariance inverse: W(t) = W(t-1) + g(t)[I - z(t)z(t)^H]W(t-1), where g(t) is a decaying gain. This enables continuous cumulant-based source separation on streaming IQ data from wideband receivers.