Independent Component Analysis (ICA)

The fundamental assumption of ICA is that the source signals are statistically independent. This means the joint probability distribution of the sources factorizes into the product of their marginal distributions: p(s₁, s₂, ..., sₙ) = p(s₁) * p(s₂) * ... * p(sₙ). The goal of the ICA algorithm is to find a linear transformation (the unmixing matrix) of the observed mixed signals that maximizes this independence. In practice, independence is measured using proxies like non-Gaussianity (via kurtosis or negentropy) or mutual information minimization.

ICA requires that at most one source signal has a Gaussian distribution. If all sources are Gaussian, the separation problem becomes impossible because the sum of independent Gaussian variables is itself Gaussian, and any orthogonal transformation (like PCA) of the mixtures preserves the Gaussian property, leaving the independent components ambiguous. ICA algorithms exploit non-Gaussianity as a measure of independence. They work by maximizing a contrast function like kurtosis (a measure of 'peakedness' or 'tailedness') or negentropy (the difference in entropy from a Gaussian). Common distributions for sources include super-Gaussian (e.g., Laplace, speech signals) or sub-Gaussian (e.g., uniform, some bio-signals).

ICA assumes the observed signals are linear, instantaneous mixtures of the independent sources. This is modeled as: x = A s, where x is the vector of observed mixed signals, s is the vector of unknown source signals, and A is an unknown mixing matrix (assumed to be square and invertible for simplicity). The model excludes:

• Time delays or convolutions (addressed by Convolutive ICA).
• Non-linear transformations (addressed by Non-linear ICA, a much harder problem). This assumption allows the problem to be framed as finding an unmixing matrix W ≈ A⁻¹ such that ŝ = W x recovers the sources.

Classical ICA assumes the number of observed mixture signals (n) is equal to the number of underlying independent sources (m). This implies the mixing matrix A is square and invertible. In practice, two scenarios exist:

• Determined Case (n = m): The standard assumption. A unique solution exists.
• Overdetermined Case (n > m): More sensors than sources. The data can first be reduced to m dimensions using PCA (whitening), then ICA is applied.
• Underdetermined Case (n < m): More sources than sensors. This is a much more difficult, ill-posed problem requiring additional constraints (e.g., sparsity) and falls under Sparse Component Analysis.

Even when assumptions hold, ICA solutions have inherent ambiguities that cannot be resolved without prior information:

• Scaling Ambiguity: The variance (energy) of each recovered source is unknown. Since both s and A are unknown, any scalar multiplier can be exchanged between a source and its corresponding column in A. Outputs are typically scaled to have unit variance.
• Permutation Ambiguity: The order of the recovered sources is unknown. The columns of A and the rows of s can be reordered identically without changing the observed mixture x. The algorithm returns sources in an arbitrary order. These ambiguities are generally acceptable for blind source separation, as the waveform shape is preserved.

While not a strict theoretical assumption, whitening (or sphering) is a critical preprocessing step for most ICA algorithms. Whitening transforms the observed data x into a new vector z = V x such that its components are uncorrelated and have unit variance (E[z zᵀ] = I). This is achieved via PCA. Whitening simplifies the ICA problem by reducing the unmixing matrix W to an orthogonal matrix (satisfying W Wᵀ = I). This reduces the number of parameters to estimate and provides a good starting point for optimization, improving convergence speed and stability. It fulfills the 'at most one Gaussian source' condition for the orthogonal transformation.

FastICA is a widely used, computationally efficient algorithm that employs a fixed-point iteration scheme to maximize non-Gaussianity as a measure of statistical independence. It uses approximations of negentropy, a robust information-theoretic measure, to find the independent components.

• Key Feature: Uses a fast, cubic-convergence Newton-like iteration.
• Optimization: Maximizes a contrast function (e.g., kurtosis or a general non-quadratic function) to find directions where the projected data is most non-Gaussian.
• Advantages: Simplicity, speed, and direct access to the independent components (no need for step-size parameters).
• Common Use: The default algorithm in many ICA software packages for general-purpose signal separation.

The Infomax algorithm formulates ICA as a problem of maximizing the mutual information between the inputs and outputs of a neural network with non-linear outputs. It is derived from the principle of maximum information preservation.

• Core Principle: Maximizes the entropy of the outputs of a non-linear transform of the estimated sources, which is equivalent to minimizing the mutual information between them.
• Implementation: Often uses a stochastic gradient ascent rule on the log-likelihood, linked to the natural gradient for faster convergence.
• Connection to Maximum Likelihood: Infomax is equivalent to maximum likelihood estimation for sources with a specified (typically super-Gaussian) prior distribution.
• Application: Historically pivotal for applications in computational neuroscience, such as separating EEG and fMRI signals.

JADE is a batch, algebraic ICA algorithm based on higher-order statistics. It performs ICA by jointly diagonalizing a set of fourth-order cumulant matrices, which contain the statistical information needed to identify the independent components.

• Methodology: First whitens the data, then finds a rotation matrix that jointly diagonalizes a set of cumulant matrices. Diagonalization corresponds to statistical independence under the cumulant matching criterion.
• Key Strength: Exploits all fourth-order information, making it statistically efficient and robust for separating sources with varying kurtosis.
• Characteristic: A tensorial method that works directly with the eigenstructure of the cumulant tensor.
• Use Case: Particularly effective for problems where a reliable algebraic solution is preferred over iterative optimization.

Kernel ICA is a general approach that uses kernel methods to measure and maximize independence. It defines independence in terms of correlations in a high-dimensional feature space (Reproducing Kernel Hilbert Space - RKHS) rather than relying on pre-defined contrast functions like kurtosis.

• Core Idea: Minimizes the correlation between nonlinear, kernel-based transformations of the estimated sources. When this correlation is zero, the variables are independent.
• Flexibility: Can capture complex, non-linear dependencies beyond second and fourth-order moments by choosing different kernel functions (e.g., Gaussian RBF).
• Advantage: Provides a very general and theoretically sound measure of independence, but can be computationally intensive.
• Application: Useful for complex, real-world signals where simpler contrast functions may fail.

Mean-Field ICA is a Bayesian approach to ICA that treats the source signals and the mixing matrix as random variables with prior distributions. It uses variational inference to approximate the posterior distribution of the sources and parameters.

• Bayesian Framework: Incorporates prior knowledge about the source distributions (e.g., Laplacian for sparsity, Gaussian mixture models) and the mixing process.
• Inference: Employs a mean-field approximation to factorize the posterior, making the computation tractable. It iteratively updates approximate distributions (q-functions) for the latent variables.
• Output: Provides not just point estimates of the sources, but full posterior distributions, enabling uncertainty quantification.
• Benefit: Robust to noise and model mismatch, and principled for handling missing data or incorporating constraints.

Projection Pursuit is a broader exploratory data analysis technique from which ICA can be derived. It seeks "interesting" low-dimensional projections of high-dimensional data, where "interesting" is defined by a projection index measuring non-Gaussianity.

• Relation to ICA: ICA can be viewed as performing projection pursuit multiple times to find N mutually orthogonal projections (the independent components) that are maximally non-Gaussian.
• Algorithmic Approach: Iteratively finds a direction that maximizes the projection index (e.g., negentropy), extracts that component, and then deflates the data to search for the next orthogonal direction.
• Contrast Functions: The choice of projection index (like those based on kurtosis or negentropy) directly links specific projection pursuit algorithms to ICA variants.
• Historical Context: Provides the statistical foundation for understanding ICA as a method for revealing the underlying structure of data.

Principal Component Analysis (PCA) is a linear dimensionality reduction technique that transforms data into a new coordinate system defined by its principal components—orthogonal directions of maximum variance. Unlike ICA, which seeks statistically independent sources, PCA finds uncorrelated (orthogonal) components. It is solved via eigenvalue decomposition of the data covariance matrix. PCA is optimal for Gaussian data and compression, while ICA is designed for separating non-Gaussian, independent sources.

Blind Source Separation (BSS) is the broader signal processing problem of recovering a set of original source signals from observed mixtures without prior knowledge of the mixing process or the sources themselves. ICA is the predominant statistical method for solving BSS when the sources are assumed to be statistically independent and non-Gaussian. Other BSS approaches include methods based on second-order statistics (like PCA for decorrelation) or sparsity assumptions.

ICA relies on the non-Gaussianity of source signals for separability. The Central Limit Theorem suggests that a mixture of independent signals will be more Gaussian than the individual sources. ICA algorithms therefore maximize a measure of non-Gaussianity to recover the sources. A common metric is kurtosis (the fourth standardized moment), which measures the 'tailedness' of a distribution. High kurtosis (super-Gaussian) and low kurtosis (sub-Gaussian) distributions are exploited by contrast functions in algorithms like FastICA.

FastICA is a highly efficient, fixed-point iterative algorithm for performing ICA. It maximizes non-Gaussianity as a measure of independence using a contrast function (often based on negentropy). Its key steps involve:

• Centering and whitening the data (similar to PCA).
• Using a fixed-point iteration scheme to find the unmixing matrix.
• Employing deflation or symmetric orthogonalization to extract multiple components. FastICA is known for its cubic convergence and is a standard benchmark due to its speed and stability.

The InfoMax principle and Maximum Likelihood Estimation (MLE) are two foundational approaches to ICA. The InfoMax algorithm maximizes the mutual information between the inputs and outputs of a neural network, which is equivalent to maximizing the independence of the outputs. The MLE approach formulates ICA as a latent variable model and maximizes the likelihood of the observed data given the unknown sources and mixing matrix. Both frameworks often lead to similar learning rules, frequently involving a nonlinearity that matches the assumed cumulative distribution function of the sources.

JADE is a notable ICA algorithm based on higher-order statistics. Instead of relying on a single non-Gaussianity measure, JADE uses cumulant matrices (specifically fourth-order cumulants). The algorithm works by:

• Whitening the observed data.
• Estimating a set of cumulant matrices.
• Finding a unitary matrix that jointly diagonalizes this set of matrices as much as possible. This diagonalization is the algebraic expression of statistical independence for the recovered components. JADE is robust but computationally more intensive than FastICA for high-dimensional data.