Probabilistic Shaping

The core engine of probabilistic shaping is the distribution matcher, a device that transforms a sequence of uniformly distributed information bits into a sequence of symbols with a target non-uniform probability distribution. This is typically implemented using constant composition distribution matching (CCDM), which generates codewords with a fixed empirical distribution. The matcher operates on blocks of symbols, ensuring that the output sequence has the exact desired symbol frequencies. The inverse operation at the receiver, the distribution dematcher, recovers the original bit stream without error. The rate loss of the matcher, which is the overhead required to achieve the target distribution, decreases as the block length increases.

In an additive white Gaussian noise channel, the capacity-achieving input distribution is itself Gaussian. Probabilistic shaping approximates this by assigning a Maxwell-Boltzmann distribution to the points of a conventional QAM constellation. This means inner constellation points, which have lower energy, are transmitted with high probability, while outer, high-energy points are used sparingly. The result is a signal with a quasi-Gaussian amplitude distribution that maximizes the mutual information between the channel input and output for a fixed average power constraint. This shaping gain can exceed 1.5 dB compared to uniform QAM at high spectral efficiencies.

A critical operational advantage of probabilistic shaping is granular rate adaptation. By simply changing the target probability distribution—controlled by a single parameter, the shaping rate—the system can achieve a continuous range of data rates without altering the underlying constellation geometry or the forward error correction code rate. This allows a transceiver to dynamically adapt to changing channel conditions with fine resolution, maintaining a constant symbol rate and baud rate. The modulation format and FEC code remain fixed, while only the distribution matcher's configuration is updated, dramatically simplifying the hardware and control plane.

Probabilistic shaping is practically realized through the probabilistic amplitude shaping (PAS) architecture, which seamlessly integrates the distribution matcher with a systematic forward error correction code. In PAS, the distribution matcher controls the amplitudes of the transmitted symbols, while the FEC encoder generates the sign bits and provides error protection. This reverse concatenation structure ensures that the shaping operation is independent of the FEC decoding, allowing the use of off-the-shelf high-performance codes like LDPC or polar codes. The dematcher at the receiver operates on the decoded bits, ensuring that any residual errors from the FEC decoder do not cause error propagation in the deshaping process.

By transmitting low-energy symbols more frequently, probabilistic shaping reduces the average energy per transmitted symbol for a given data rate. This directly translates to improved power efficiency and a lower required signal-to-noise ratio (SNR) to achieve a target bit-error rate. In optical and wireless systems, this shaping gain can be traded for extended reach, reduced amplifier power, or higher tolerance to non-linear impairments. The peak-to-average power ratio (PAPR) of the shaped signal is also modified, which must be managed in systems with non-linear power amplifiers, but the net system margin improvement is substantial.

In fiber-optic communications, the Gaussian-like distribution of probabilistically shaped signals exhibits improved tolerance to fiber non-linearities, particularly the Kerr effect. The reduced probability of high-amplitude symbols decreases the non-linear phase noise accumulated during propagation. This effect, sometimes called non-linear shaping gain, is an additional benefit beyond the linear AWGN shaping gain. The optimized distribution reduces the amplitude-dependent self-phase modulation, allowing for higher launch powers and longer transmission distances before non-linear distortion becomes the limiting factor. This makes probabilistic shaping a key enabler for next-generation coherent optical transceivers operating at 800G and beyond.

The core algorithmic engine that transforms a sequence of uniformly distributed information bits into a sequence of constellation symbols with a target non-uniform (typically Maxwell-Boltzmann) distribution. This is the physical layer component that realizes probabilistic shaping.

• Constant Composition DM (CCDM): Uses arithmetic coding to generate fixed-length output blocks with an exact empirical distribution, enabling fixed-rate transmission.
• Enumerative Sphere Shaping (ESS): Maps bits to sequences within a multi-dimensional sphere, achieving near-optimal shaping gain with lower complexity than CCDM for long block lengths.
• Rate Loss: The penalty in spectral efficiency incurred because the DM must map discrete bits to a discrete set of sequences, vanishing as block length increases.

The optimal probability mass function for constellation points on an additive white Gaussian noise (AWGN) channel when the only constraint is average power. It assigns exponentially decreasing probability to symbols with higher energy.

• Functional Form: P(x) ∝ exp(-ν|x|²), where ν is a shaping parameter controlling the trade-off between entropy and average power.
• Shaping Gain: The ultimate limit is 1.53 dB, the difference between the capacity of a uniform input and the true AWGN channel capacity. Practical systems achieve 0.5–1.2 dB.
• Geometric vs. Probabilistic: Pure geometric shaping moves points; probabilistic shaping changes their frequencies. The optimal approach is a joint optimization of both.

A specific architectural framework that combines a distribution matcher with a systematic FEC code. The DM shapes the amplitudes of the constellation, and the FEC code adds the sign bits, which are uniformly distributed.

• Amplitude-Sign Factorization: For a square QAM constellation, the real and imaginary parts are shaped independently. The DM generates non-uniform amplitude levels; the FEC parity bits determine the quadrant (sign).
• Systematic Encoding: The information bits are the shaped amplitude bits. The FEC encoder computes parity bits that are mapped to the sign, preserving the amplitude distribution.
• 5G NR Rel. 16+: PAS is the foundational architecture for the probabilistic shaping adopted in modern optical and wireless standards.

A learned alternative to algorithmic distribution matchers, using a neural network to directly map uniform bits to shaped symbols. This approach can learn shaping distributions for arbitrary, non-AWGN channels where the optimal distribution is not analytically known.

• Normalizing Flows: Invertible neural networks that transform a simple uniform distribution into a complex target distribution, enabling exact likelihood computation and low-complexity inverse mapping at the receiver.
• Joint Geometric and Probabilistic Shaping: A single neural network can simultaneously optimize the positions and probabilities of constellation points, learning a joint shaping scheme that outperforms separate optimization.
• Channel-Agnostic Training: The neural DM can be trained on a differentiable channel model or directly on measured channel data, learning the optimal shaping strategy for non-linear fiber or hardware-impaired channels.