Maximum Likelihood Sequence Estimation (MLSE)

MLSE operates fundamentally differently from symbol-by-symbol detectors. Instead of making independent decisions on each received symbol, it evaluates entire candidate sequences against the received signal. The detector selects the sequence ŝ that maximizes the conditional probability density function p(r|s), where r is the received signal vector and s is a candidate transmitted sequence. This joint optimization over the full sequence length exploits the memory structure introduced by the channel, making it the maximum a posteriori (MAP) sequence detector when all sequences are equally probable.

• Metric: Maximizes the log-likelihood function over the sequence
• Memory exploitation: Uses inter-symbol interference (ISI) as information rather than treating it as noise
• Optimality: Achieves the theoretical minimum sequence error probability for channels with known memory

The Viterbi algorithm provides a computationally efficient recursive implementation of MLSE, avoiding the exponential complexity of exhaustive sequence comparison. It operates on a trellis diagram representing the channel's finite-state machine, where each state corresponds to the channel memory contents. At each time step, the algorithm computes branch metrics (the squared Euclidean distance between the received sample and the hypothesized noiseless signal) and accumulates them into path metrics.

• Complexity: O(M^L) per symbol, where M is the constellation size and L is the channel memory length
• Survivor paths: Only one path per state is retained at each trellis stage
• Traceback depth: Typically 5-10 times the channel memory length for negligible truncation loss
• Add-Compare-Select (ACS): The core butterfly operation performed at each trellis node

MLSE requires an accurate model of the dispersive channel that introduces memory into the received signal. The channel is typically represented as a finite impulse response (FIR) filter with coefficients h = [h₀, h₁, ..., h_L], where L is the memory length in symbol periods. The received sample at time k is: r_k = Σ h_i · s_{k-i} + n_k, where n_k is additive noise. This convolution creates a trellis with M^L states, each representing a possible combination of the L most recent transmitted symbols.

• Channel estimation: MLSE performance critically depends on accurate channel impulse response knowledge
• Adaptive MLSE: Uses decision-directed or pilot-aided channel tracking for time-varying channels
• Whitened matched filter: Preprocessing step that shapes the channel response to a causal, minimum-phase form suitable for the Viterbi detector

In the context of automatic modulation classification (AMC), MLSE serves a dual role. First, it can be used as the optimal demodulator within a likelihood-based classifier that evaluates the likelihood function under each modulation hypothesis. Second, the accumulated path metrics from the Viterbi algorithm provide a natural sequence likelihood score that can be compared across candidate modulation formats. The classifier selects the modulation scheme whose MLSE demodulator produces the highest likelihood for the received sequence.

• Composite hypothesis testing: MLSE handles unknown data symbols as nuisance parameters by maximizing over them
• GLRT integration: The generalized likelihood ratio test uses MLSE to estimate the transmitted sequence under each modulation hypothesis
• Per-survivor processing: Enables joint channel estimation and sequence detection for blind modulation classification scenarios

The probability of sequence error for MLSE is dominated by error events—specific divergences from the correct path through the trellis that begin and end at common states. The pairwise error probability between two sequences differing over an error event of length d is upper-bounded by: P(error) ≤ Q(d_min / 2σ), where d_min is the minimum Euclidean distance between any two distinct sequences and σ² is the noise variance. This distance depends on both the channel response and the modulation constellation.

• Distance spectrum: The set of all error event distances and their multiplicities determines MLSE performance
• Coding gain: MLSE over ISI channels can provide gains comparable to convolutional coding over AWGN channels
• Error floors: Residual ISI from imperfect channel estimation or finite traceback depth creates irreducible error floors at high SNR

Despite its optimality, MLSE faces practical constraints that limit its direct application in many scenarios. The computational complexity grows exponentially with channel memory length, making it impractical for highly dispersive channels without complexity reduction techniques. Additionally, MLSE requires coherent detection with accurate carrier and timing synchronization, as phase errors distort the Euclidean distance metrics used in the Viterbi algorithm.

• Reduced-state techniques: Decision-feedback sequence estimation (DFSE) and reduced-state sequence estimation (RSSE) trade performance for tractable complexity
• Soft-output variants: Soft-output Viterbi algorithm (SOVA) provides reliability information for concatenated systems
• Synchronization sensitivity: Phase and timing offsets must be estimated and compensated before MLSE processing
• Alternative: Linear equalization or decision-feedback equalization for complexity-constrained applications