T-Distributed Stochastic Neighbor Embedding (t-SNE)

t-SNE converts high-dimensional Euclidean distances between feature vectors (e.g., cumulants or spectral features) into conditional probabilities representing similarity. It then constructs a similar probability distribution in the low-dimensional map. The algorithm minimizes the Kullback-Leibler (KL) divergence between these two distributions, ensuring that points that are neighbors in the original feature space remain neighbors in the visualization. This is fundamentally different from linear methods like PCA, which focus on preserving large pairwise distances.

In the low-dimensional embedding space, t-SNE uses a Student's t-distribution with one degree of freedom (a Cauchy distribution) instead of a Gaussian. This heavy-tailed distribution alleviates the crowding problem inherent in dimensionality reduction. It allows moderately distant points in the high-dimensional space to be modeled by larger distances in the map, preventing them from collapsing into a single point and creating more distinct, well-separated clusters for visual analysis of modulation types.

The perplexity parameter is a smooth measure of the effective number of neighbors considered for each data point. It is defined as (2^{H(P_i)}), where (H(P_i)) is the Shannon entropy of the conditional probability distribution. Typical values range from 5 to 50. A low perplexity focuses on very local structure, potentially fragmenting a modulation cluster into sub-clusters. A high perplexity captures more global structure but may merge distinct modulation types. Tuning perplexity is critical for revealing the true Voronoi region separability.

t-SNE minimizes the non-convex KL divergence cost function using gradient descent with momentum. The gradient has a physical interpretation: an attractive force pulls neighbors together, while a repulsive force pushes all points apart, scaled by the t-distribution. The optimization is sensitive to early exaggeration, a phase where attractive forces are amplified to create widely separated clusters. This non-linear optimization is stochastic, meaning different runs can produce different maps, so multiple initializations are often required to confirm the stability of modulation clusters.

In automatic modulation classification, t-SNE is applied to feature vectors derived from IQ samples or higher-order cumulants. The resulting 2D or 3D plot provides an intuitive visual diagnostic of classifier feasibility. Clear, non-overlapping clusters indicate that the chosen features (e.g., ring ratio for APSK or spectral correlation density patterns) are discriminative. Overlapping clusters suggest that the features are insufficient to separate those modulation schemes, guiding feature engineering before training a deep learning model.

t-SNE is a non-parametric method, meaning it does not learn an explicit mapping function from the high-dimensional space to the low-dimensional space. It only produces an embedding for the data it was trained on. To map new, unseen signal samples, the entire algorithm must be re-run, which is computationally prohibitive for real-time spectrum classification. Parametric variants or alternative techniques like UMAP are often preferred when a reusable transform is required for streaming IQ data in deployed systems.

A linear dimensionality reduction technique that projects high-dimensional constellation feature vectors onto orthogonal axes of maximum variance. Unlike t-SNE, PCA preserves global Euclidean distances between IQ sample clusters but often fails to separate non-linear modulation manifolds. It is computationally faster and deterministic, making it suitable for initial exploratory analysis before applying t-SNE. PCA is frequently used to whiten features prior to t-SNE embedding to reduce noise and accelerate convergence.

A manifold learning technique that competes with t-SNE for visualizing modulation clusters. UMAP constructs a fuzzy topological representation of the high-dimensional feature space and optimizes a low-dimensional embedding to preserve both local and global structure. It typically runs faster than t-SNE on large datasets of IQ sample features and better preserves the global relationships between different modulation formats, making it preferable for datasets with many distinct signal types.

A critical t-SNE parameter that balances attention between local and global aspects of the constellation feature space. Perplexity is loosely interpreted as the number of effective nearest neighbors considered for each data point. For modulation recognition datasets:

• Low values (5-15): Reveal fine-grained cluster substructure but may fragment a single modulation type
• High values (30-50): Capture broader relationships between different modulation families
• Typical defaults of 30 often fail on sparse higher-order cumulant features

The objective function minimized by t-SNE, measuring the divergence between two probability distributions: the pairwise similarities in the high-dimensional signal feature space and the pairwise similarities in the low-dimensional embedding. The algorithm uses a heavy-tailed Student's t-distribution in the low-dimensional space to alleviate the crowding problem, preventing distant modulation clusters from collapsing onto each other during optimization.

Before applying t-SNE to signal constellation data, raw IQ samples must be transformed into discriminative feature vectors. Common preprocessing pipelines include:

• Computing higher-order cumulants (C20, C21, C40, C41, C42, C60, C63)
• Extracting cyclostationary signatures via spectral correlation density
• Calculating instantaneous amplitude, phase, and frequency statistics
• Using the bottleneck activations of a pre-trained deep learning modulation classifier The quality of these features directly determines cluster separability in the t-SNE plot.

Quantitative measures used to assess the quality of t-SNE embeddings for modulation format separability:

• Silhouette Score: Measures how similar a point is to its own cluster versus other clusters, ranging from -1 to 1
• Davies-Bouldin Index: The average similarity between each cluster and its most similar counterpart; lower values indicate better separation
• Trustworthiness: Measures the proportion of points that are neighbors in the high-dimensional space and remain neighbors in the embedding These metrics provide objective validation beyond visual inspection of constellation cluster plots.