Maximum Likelihood Sequence Estimation (MLSE)

MLSE fundamentally differs from symbol-by-symbol equalizers by making decisions on entire sequences rather than isolated symbols. The detector considers all possible transmitted sequences and selects the one that maximizes the likelihood function given the received signal. This holistic approach exploits the memory inherent in ISI channels, where each received sample contains energy from multiple adjacent symbols. By jointly decoding the sequence, MLSE achieves the theoretical minimum error probability for channels with known impulse responses.

The practical implementation of MLSE relies on the Viterbi algorithm, a dynamic programming technique that efficiently searches the trellis of channel states without evaluating every possible sequence exhaustively. Key aspects include:

• Trellis construction: States represent the last L-1 transmitted symbols, where L is the channel memory length
• Branch metric computation: Each transition's cost is the squared Euclidean distance between the received sample and the noiseless channel output
• Add-Compare-Select (ACS) operations: The core recursive step that eliminates suboptimal paths at each trellis stage
• Traceback: After processing the sequence, the surviving path with minimum cumulative metric is traced back to recover the ML estimate

MLSE requires accurate knowledge of the channel impulse response to compute branch metrics. In practice, this necessitates a companion channel estimation block that provides the equalizer with tap coefficients. The estimation can be:

• Training-based: Using known pilot sequences multiplexed into the transmission
• Decision-directed: Updating estimates from previously detected symbols during tracking mode
• Blind: Deriving channel parameters from received signal statistics without overhead Estimation errors propagate directly into metric computation, making robust channel tracking critical for maintaining near-optimal performance in time-varying environments.

The complexity of MLSE grows exponentially with channel memory length L. For a modulation with M states, the trellis contains M^(L-1) states, making full MLSE impractical for channels with long delay spreads. Mitigation strategies include:

• Delayed decision-feedback sequence estimation (DDFSE): Truncating the channel memory and using decision feedback for residual ISI
• Reduced-state sequence estimation (RSSE): Merging trellis states using set partitioning principles
• Per-survivor processing (PSP): Embedding adaptive channel estimation within each survivor path These suboptimal variants trade marginal performance loss for dramatic complexity reduction, enabling real-time implementation on FPGA and ASIC platforms.

Modern communication systems pair MLSE with forward error correction (FEC) codes, requiring soft reliability information rather than hard decisions. The Soft-Output Viterbi Algorithm (SOVA) extends MLSE to produce log-likelihood ratios (LLRs) for each bit. Alternatively, the BCJR algorithm (Bahl-Cocke-Jelinek-Raviv) computes exact a posteriori probabilities for each symbol, forming the foundation of turbo equalization. In turbo equalization, soft information iterates between the equalizer and channel decoder, progressively refining estimates until convergence.

MLSE and its reduced-complexity variants are deployed in numerous wireless communication standards where ISI is the dominant impairment:

• GSM/EDGE: MLSE equalization is mandatory for handling the severe multipath in 200 kHz narrowband channels
• Bluetooth EDR: Sequence estimation improves performance of π/4-DQPSK and 8DPSK modes
• Underwater acoustic communications: Long delay spreads necessitate MLSE-based receivers
• Magnetic recording channels: Partial-response maximum likelihood (PRML) is a specialized MLSE form for storage systems In 5G and beyond, MLSE principles are integrated into advanced receivers for single-carrier frequency-domain equalization (SC-FDE) systems.