Geometric Shaping

Unlike traditional Quadrature Amplitude Modulation (QAM) where points are fixed on a rigid rectangular grid, geometric shaping treats the location of each constellation point as a continuous variable to be optimized. The objective is to maximize the generalized mutual information (GMI) or minimize the bit error rate (BER) for a specific signal-to-noise ratio (SNR) and channel model, such as a non-linear fiber optic link. This results in non-uniform, cloud-like arrangements that more closely approach the Shannon capacity limit than regular lattice structures.

Geometric shaping is particularly powerful for channels with non-linear impairments, such as satellite transponders operating near saturation or long-haul optical fibers. In these scenarios, the optimal constellation is no longer Gaussian-like. The shaping algorithm can be designed to pre-compensate for the specific amplitude-to-amplitude (AM/AM) and amplitude-to-phase (AM/PM) distortion of the channel. By warping the constellation geometry, the received signal points can be made to land on a regular grid after passing through the non-linearity, dramatically simplifying the receiver's digital signal processing.

Geometric shaping is often compared to its counterpart, probabilistic shaping. The key distinction lies in the mechanism:

• Geometric Shaping: Uses a uniform probability distribution over a non-uniformly spaced set of points. The locations are optimized.
• Probabilistic Shaping: Uses a non-uniform probability distribution (e.g., Maxwell-Boltzmann) over uniformly spaced points (a regular QAM lattice). The probabilities are optimized. Hybrid approaches that combine both geometric and probabilistic shaping can achieve the ultimate capacity-approaching performance.

A modern approach to geometric shaping uses an autoencoder neural network to learn the optimal constellation geometry directly from data. The transmitter is modeled as an encoder network that maps bits to complex symbols, the channel is a non-trainable stochastic layer, and the receiver is a decoder network. By training the system end-to-end to minimize the symbol error rate, the network discovers highly efficient, channel-specific geometric arrangements that often outperform analytically derived constellations, especially for complex or unknown channel models.

Classic iterative algorithms for geometric shaping, such as the Lloyd-Max algorithm or gradient descent on the error probability, work by moving points to reduce mutual interference. The process involves:

• Calculating the Voronoi regions for the current point set.
• Computing the centroid of the conditional probability distribution within each region.
• Moving the constellation point to that centroid. This is repeated until convergence, effectively minimizing the mean squared error for a given noise distribution.

The ultimate goal of geometric shaping is to close the gap to the Shannon limit. For the additive white Gaussian noise (AWGN) channel, the capacity-achieving input distribution is Gaussian. A geometrically shaped constellation approximates this continuous Gaussian distribution with a discrete set of points. By optimizing the placement of 64 or 256 points in a non-uniform, spherical arrangement, a system can achieve a shaping gain of up to 1.53 dB over a uniform square QAM constellation, representing a significant increase in power efficiency.

The complementary technique to geometric shaping that achieves capacity gains by transmitting constellation points with non-uniform probability. Inner, lower-energy points are used more frequently than outer points, creating a Gaussian-like distribution that minimizes average power for a given data rate. While geometric shaping optimizes the locations of points, probabilistic shaping optimizes their frequency of use, and the two can be combined for ultimate performance.

The traditional baseline that geometric shaping seeks to improve upon. QAM arranges points on a regular rectangular grid, which is optimal only for additive white Gaussian noise (AWGN) channels. In non-linear fiber channels, this rigid structure is suboptimal. Geometric shaping breaks free from this lattice constraint, allowing points to migrate to positions that better withstand the specific impairments of the optical path.

A modulation format that arranges points on multiple concentric rings, offering a peak-to-average power ratio (PAPR) advantage over square QAM. APSK can be seen as a constrained form of geometric shaping where the optimization is limited to finding the optimal number of rings, their radii, and the number of points per ring. True geometric shaping removes even these constraints for a fully unconstrained optimization.

An unsupervised machine learning algorithm that can be used to blindly discover an optimal constellation from received data. By treating the received IQ samples as data points, K-means iteratively finds the centroid positions that minimize the within-cluster sum of squares. This effectively performs geometric shaping in a model-free way, adapting the constellation to the empirical channel impairments without requiring an explicit channel model.

The primary quality metric that geometric shaping aims to minimize. EVM measures the Euclidean distance between the received symbol and its ideal reference point. A well-shaped constellation will exhibit lower EVM under non-linear distortion than a regular QAM constellation at the same power, as its points are pre-compensated for the channel's warping effects, leading to a tighter, more discernible clustering at the receiver.

The ultimate objective function for geometric shaping. Rather than minimizing symbol error rate, advanced shaping algorithms directly maximize the mutual information (MI) between the transmitted and received signals. This is the true measure of the information rate that can be reliably communicated. A constellation optimized for MI will often have an irregular, cloud-like appearance, as it prioritizes information throughput over geometric regularity.