Gaussian Mixture Model (GMM)

Unlike hard clustering algorithms like K-Means that assign each IQ sample to a single constellation point, GMMs provide a soft assignment. Each sample is assigned a posterior probability of belonging to every Gaussian component. This quantifies the uncertainty of a received symbol being a specific constellation point, which is critical for soft-decision forward error correction decoders that use log-likelihood ratios to improve bit error rate performance.

GMM parameters are typically estimated using the Expectation-Maximization (EM) algorithm, an iterative two-step process:

• E-Step (Expectation): Calculates the posterior probability (responsibility) that each Gaussian component generated each observed IQ sample, given the current parameter estimates.
• M-Step (Maximization): Updates the component means, covariances, and mixing weights to maximize the expected log-likelihood computed in the E-step. This process repeats until convergence, monotonically increasing the likelihood of the data under the model.

A key advantage of GMMs over simpler centroid-based methods is the ability to model the full covariance structure of each constellation point cluster. The covariance matrix captures the spread and orientation of noise in the IQ plane. A full covariance matrix can model elliptical, rotated clusters caused by IQ imbalance, while a tied covariance structure assumes all components share the same noise characteristics, useful for modeling pure additive white Gaussian noise (AWGN).

Each Gaussian component in the mixture has an associated mixing weight (π_k). In the context of constellation modeling, this weight represents the prior probability of a specific constellation point being transmitted. For standard uniform modulation formats like QPSK or 16-QAM, these weights are expected to be equal. Detecting a significant deviation from a uniform distribution can indicate probabilistic shaping, where outer constellation points are transmitted less frequently to improve energy efficiency.

The number of Gaussian components (K) must be chosen to match the number of points in the unknown constellation. Model selection criteria like the Bayesian Information Criterion (BIC) and Akaike Information Criterion (AIC) are used to select the optimal K. These metrics balance the goodness-of-fit (log-likelihood) against model complexity (number of parameters), penalizing overfitting. A sharp elbow in the BIC curve as K increases often reveals the true modulation order, enabling blind modulation identification.

The EM algorithm is sensitive to initialization and can converge to poor local maxima. A standard practice is to initialize the GMM component means using the K-Means++ algorithm, which selects initial centroids that are well-separated in the IQ plane. The initial covariances are set to the empirical covariance of the samples assigned to each cluster, and mixing weights are proportional to cluster sizes. This provides a data-driven starting point that significantly accelerates EM convergence and avoids degenerate solutions.

The iterative optimization engine that drives GMM parameter estimation. The E-step computes the posterior probability (responsibility) that each IQ sample belongs to a specific Gaussian component. The M-step re-estimates the mean, covariance, and mixing weight of each component by maximizing the expected log-likelihood. This guarantees convergence to a local maximum, making it the standard solver for blind constellation reconstruction when the modulation format is unknown.

Unlike K-Means or Voronoi partitioning, which assign a received symbol to exactly one constellation point, a GMM provides a soft assignment—a probability vector over all possible symbols. This is critical for modern decoders using soft-decision forward error correction. The log-likelihood ratios (LLRs) derived from the GMM's posterior probabilities directly feed into LDPC or turbo decoders, capturing the channel's uncertainty rather than discarding it with a premature hard decision.

Each Gaussian component in a GMM is defined by a full covariance matrix that models the shape and orientation of the noise cloud around a constellation point. This captures real-world impairments that a spherical model misses:

• IQ Imbalance: Manifests as elliptical, tilted clusters.
• Phase Noise: Creates an arc-shaped spread tangential to the constellation circle.
• Correlated Noise: Appears when filtering introduces dependency between I and Q samples. Constraining the covariance structure (e.g., diagonal, tied, or spherical) trades model fidelity for estimation robustness with limited data.

Determining the correct number of Gaussian components (K) is the central challenge in blind modulation recognition. The GMM must discover the constellation cardinality without prior knowledge. Information-theoretic criteria penalize model complexity:

• Bayesian Information Criterion (BIC): Applies a penalty proportional to log(N) for each parameter, favoring simpler models with large sample sizes.
• Akaike Information Criterion (AIC): Uses a lighter penalty, making it less conservative than BIC.
• Minimum Description Length (MDL): Selects the model that provides the shortest lossless encoding of the data.

The EM algorithm is sensitive to initialization and can converge to poor local maxima. Effective strategies for seeding GMM components in the IQ plane include:

• K-Means++: Selects initial centroids with probability proportional to their squared distance from existing centroids, ensuring good coverage of the constellation.
• Hierarchical Agglomerative Clustering: Builds a dendrogram of IQ samples and cuts it at the desired number of clusters, robust for irregular constellations like APSK.
• Subtractive Clustering: Identifies dense regions in the scatter plot as initial component means, effective when the number of clusters is unknown.

A GMM trained on received IQ samples produces a generative model of the signal. The average log-likelihood of the data under each candidate modulation hypothesis (e.g., QPSK, 16-QAM, 64-QAM) serves as a classification score. The modulation format that maximizes this likelihood is selected. This is the optimal classifier in the Bayesian sense when the true channel distribution matches the GMM assumption, outperforming feature-based methods like cumulant thresholding at low SNR.