End-to-End Autoencoder

The defining characteristic of an end-to-end autoencoder is the co-optimization of the transmitter and receiver as a single neural network. Unlike classical systems where modulation, coding, and equalization are designed in isolation, the entire mapping from source bits s to decoded bits ŝ is learned simultaneously.

• Single Loss Function: A single cross-entropy or mean-squared error loss backpropagates from the receiver output through the channel model to the transmitter.
• No Expert Knowledge: The system discovers optimal constellations and coding strategies without human-designed alphabets like QAM.
• Hardware Impairment Compensation: The joint optimization naturally learns to pre-distort signals to compensate for known non-linearities in the RF front-end.

A critical enabler for gradient-based learning is a differentiable channel model that sits between the transmitter and receiver during training. This model must allow gradients to flow from the receiver loss back to the transmitter weights.

• Stochastic Regularization: The channel is modeled as a probabilistic layer with trainable or fixed noise parameters (e.g., additive white Gaussian noise, phase noise).
• Generative Adversarial Networks (GANs): When a mathematical model is unavailable, a GAN can learn a differentiable surrogate from recorded channel measurements.
• Mixture of Experts: Complex channels can be approximated by a weighted combination of simpler, differentiable sub-models.

The transmitter's final layer learns a geometric constellation in the I/Q plane that maximizes mutual information for the specific channel. This moves beyond fixed, square QAM grids to irregular, data-driven arrangements.

• Geometric Shaping: The network learns the optimal spatial positions of constellation points, often forming non-uniform, circular, or lattice structures.
• Probabilistic Shaping: Combined with a distribution matcher, the network learns to transmit low-energy points more frequently, approaching the Shannon capacity limit.
• Robustness: Learned constellations are inherently optimized for the specific signal-to-noise ratio (SNR) and impairment regime used during training.

Advanced autoencoder architectures can operate without explicit pilot symbols or a separate channel estimation block. The receiver learns to perform blind detection by exploiting structure in the learned transmit waveform.

• Superimposed Pilots: Training symbols are added to the data signal rather than occupying dedicated time-frequency resources, improving spectral efficiency.
• Non-Coherent Autoencoder: The receiver is trained to be invariant to unknown channel phase and amplitude, learning a robust representation that does not require CSI.
• Joint Detection and Estimation: The neural receiver simultaneously performs channel equalization and symbol demapping in a single inference pass.

Pure black-box neural networks can be sample-inefficient. Model-based autoencoders integrate known algorithmic structures as non-trainable layers, combining the efficiency of expert knowledge with the adaptability of deep learning.

• Algorithm Unfolding: Classical iterative algorithms like the Viterbi or belief propagation are unfolded into a neural network with learnable parameters.
• FFT/IFFT Layers: The Fast Fourier Transform is integrated as a fixed layer to efficiently generate OFDM waveforms, while the network learns optimal symbol mapping.
• ViterbiNet: A receiver that replaces the hand-crafted branch metric in the Viterbi algorithm with a learned neural network, enabling decoding over unknown channel memory.

For many applications, perfect bit recovery is unnecessary. Task-oriented communication optimizes the transceiver to preserve only the semantic information relevant to a downstream inference task, such as image classification.

• Joint Source-Channel Coding (JSCC): A single network maps raw sensor data (e.g., pixels) directly to channel symbols, bypassing separate source and channel codecs.
• Information Bottleneck: The encoder learns a compressed, stochastic representation that is maximally informative about the target task, not the raw input.
• Bandwidth Efficiency: By transmitting only task-relevant features, these systems can operate reliably at extremely low signal-to-noise ratios where classical systems fail.