Geometric Constellation Shaping

Unlike probabilistic shaping which adjusts point probabilities, GCS treats the physical (I, Q) coordinates of each constellation point as a trainable parameter. A neural network or direct optimization loop uses stochastic gradient descent to move points in the complex plane, maximizing the Generalized Mutual Information (GMI) for a specific Signal-to-Noise Ratio (SNR).

• Mechanism: Backpropagation through the channel model adjusts geometry.
• Result: Non-uniform, Gaussian-like distributions that approach the Shannon limit.
• Key Metric: Maximizes bits/symbol for a given channel condition.

GCS is often implemented as an autoencoder neural network representing the entire physical layer. The transmitter network maps bits to optimized constellation points, the middle layers simulate the stochastic channel (AWGN, fading), and the receiver network learns to demodulate.

• Joint Optimization: Transmitter geometry and receiver decision boundaries are learned simultaneously.
• Channel-Agnostic: The same architecture adapts to fiber, wireless, or satellite channels.
• Hardware Awareness: Non-linearities like power amplifier compression can be included in the training loop.

The optimized constellations abandon rigid square grids. The resulting geometries often exhibit hexagonal packing in dense regions and sparse outer points to handle high-amplitude symbols efficiently.

• Gaussian-like Envelope: Point density follows a near-Gaussian distribution in the complex plane.
• Irregular Boundaries: No enforced symmetry; the optimizer finds the most efficient packing.
• Implementation: Look-up tables (LUTs) store the final optimized coordinates for low-complexity mapping.

A GCS constellation is trained for a specific operating SNR. Unlike universal QAM, the geometry morphs to match the channel's noise profile. This creates a family of constellations, each optimal at a distinct SNR point.

• Adaptive Modulation: A system can switch between pre-optimized GCS constellations as channel conditions change.
• Non-Linear Channels: GCS excels in fiber-optic systems where the Kerr non-linearity dominates, learning to pre-distort the geometry.
• Training Data: Requires a differentiable channel model or extensive real-world measurements.

The loss function for GCS is not the Bit Error Rate (BER) directly, but a differentiable surrogate: Mutual Information (MI) or its generalized version (GMI). This metric measures the total information reliably transmitted per symbol.

• GMI as Loss: Maximizing GMI directly optimizes the achievable information rate.
• Bit-Metric Decoding: GMI accounts for practical bit-interleaved coded modulation (BICM) systems.
• Gradient Estimation: Techniques like the REINFORCE algorithm or reparameterization tricks provide low-variance gradient estimates through the stochastic channel.

GCS can be combined with Probabilistic Constellation Shaping (PCS) for ultimate performance. The geometry is first optimized (GCS), and then a distribution matcher assigns non-uniform probabilities to the geometrically shaped points.

• Hybrid Shaping: Achieves shaping gains exceeding either technique alone.
• Fine-Grained Rate Adaptation: PCS provides flexible rate tuning, while GCS provides the optimal geometric envelope.
• Standardization: This hybrid approach is a key enabler for next-generation coherent optical transceivers operating at 800G and beyond.

The dual approach to geometric shaping. Instead of moving constellation points, PCS assigns non-uniform probabilities to fixed QAM points, transmitting outer points less frequently. This creates a Gaussian-like distribution that maximizes mutual information. Often implemented via a distribution matcher using constant composition codes. Geometric shaping typically outperforms PCS at lower SNR, while PCS excels at high SNR with simpler demapping.

The core objective function for GCS. Since mutual information between transmitted symbols and received signals has no closed form in complex channels, neural estimators like MINE or DONSER are used. These train a discriminator network to approximate the Kullback-Leibler divergence between joint and marginal distributions, providing a differentiable loss for gradient descent on constellation points.

A practical, achievable rate metric used when a bit-metric decoder is employed. GMI accounts for the suboptimality of treating bits independently after demapping. GCS often maximizes GMI rather than pure mutual information to optimize for real-world LDPC or Polar-coded systems. The bit labeling becomes a joint optimization variable alongside point positions.

The historical precursor to neural GCS, used in DVB-S2X and ATSC 3.0 broadcast standards. These constellations were hand-crafted or optimized via simulated annealing for specific SNR ranges. Modern GCS extends this by using gradient descent through a differentiable channel model, enabling optimization for arbitrary channel distributions including non-linear and fading scenarios.

A complementary technique where an autoencoder compresses Channel State Information at the receiver into a low-dimensional latent code. This code is fed back to the transmitter, which can then select or adapt the optimal geometrically shaped constellation for the current channel condition. Together, they form a closed-loop adaptive modulation system.