Hidden Markov Model (HMM)

An HMM couples two distinct probability mechanisms: a hidden state transition process and an observable emission process. The state sequence evolves according to a Markov chain, meaning the probability of the next state depends only on the current state, not the full history. At each time step, the hidden state emits an observation according to a state-specific probability distribution. This duality allows HMMs to model systems where the underlying mechanism is never directly visible—such as a channel's fading state or a transmitter's internal mode—while the received signal samples serve as the observable emissions.

Every HMM application reduces to solving one or more of three fundamental problems:

• Evaluation: Given a model and an observation sequence, compute the probability that the model generated that sequence. Solved by the Forward-Backward algorithm.
• Decoding: Given a model and observations, find the most likely hidden state sequence. Solved by the Viterbi algorithm.
• Learning: Given an observation sequence and a model structure, estimate the parameters that maximize the likelihood of the data. Solved by the Baum-Welch algorithm, a special case of Expectation-Maximization.

The Markov property enforces a finite memory: the future is conditionally independent of the past given the present state. This is both a strength and a limitation. It enables tractable inference via dynamic programming but restricts the model's ability to capture long-range dependencies directly. In modulation classification, this maps naturally to channels with finite-duration intersymbol interference or fading with a defined coherence time. The state space cardinality—the number of distinct hidden states—determines the model's representational capacity and computational cost.

An HMM is fully specified by three parameter matrices:

• A (Transition Matrix): a_ij = P(state j at t+1 | state i at t), defining the hidden dynamics.
• B (Emission Matrix): b_j(k) = P(observing symbol k | state j), linking hidden states to observations.
• π (Initial State Distribution): π_i = P(state i at time 1), seeding the process. For continuous observations common in signal processing, B is replaced by parametric probability density functions—typically Gaussian mixtures—whose parameters are learned alongside the transition probabilities. This compact parameterization makes HMMs statistically efficient.

Brute-force evaluation of an HMM over a sequence of length T with N states requires O(N^T) operations—computationally intractable. HMMs exploit the Markov property through dynamic programming to reduce this to O(N²T). The Forward algorithm computes the likelihood by recursively summing over all paths, while the Viterbi algorithm finds the single best path by recursively maximizing. This efficiency is what makes HMMs practical for real-time modulation classification tasks where latency constraints demand rapid inference on streaming IQ samples.

HMMs are fundamentally generative models: they learn the joint probability distribution P(X, Y) of observations and hidden states. This contrasts with discriminative classifiers like neural networks that directly model the decision boundary P(Y|X). The generative approach offers advantages in modulation classification:

• Robustness to missing data: can marginalize over unobserved samples.
• Natural handling of variable-length sequences: no fixed input dimensionality.
• Interpretability: learned parameters correspond to physically meaningful quantities like channel transition probabilities and noise characteristics.

HMMs capture the temporal dependencies introduced by wireless channels with memory, such as those exhibiting burst errors or fading correlations. Unlike memoryless classifiers that treat each symbol independently, HMM-based approaches model the state transition probabilities of the channel, enabling more accurate likelihood computation for sequential modulation detection.

• Models Gilbert-Elliott channels with distinct good/bad states
• Captures Markovian fading where channel gain evolves according to a stochastic process
• Improves classification accuracy in slow-fading environments by 15-30% over memoryless methods

The Baum-Welch algorithm, a specialized case of the Expectation-Maximization (EM) procedure, iteratively estimates HMM parameters when the underlying state sequence is unknown. In modulation classification, this enables blind estimation of transition matrices and emission probabilities directly from the received IQ samples.

• E-step: Computes forward-backward probabilities to estimate hidden state occupancy
• M-step: Updates model parameters to maximize the expected log-likelihood
• Converges to a local maximum of the likelihood function, with initialization sensitivity requiring multiple random restarts

The Viterbi algorithm finds the most probable hidden state sequence given observations, enabling maximum likelihood sequence estimation (MLSE) for modulation classification. By decoding the most likely state path through the trellis, classifiers can identify modulation schemes that exhibit finite-state Markov structure.

• Computes the shortest path through a weighted trellis diagram in O(T × N²) time
• Used for classifying continuous phase modulation (CPM) signals with inherent memory
• Enables joint channel estimation and modulation recognition by treating the channel as a hidden state

The emission probability defines how hidden states generate observable IQ samples. For modulation classification, emission models must capture the conditional distribution of received symbols given the modulation state and channel condition.

• Gaussian mixture models (GMMs) approximate arbitrary emission densities for non-Gaussian interference
• Discrete symbol emissions map directly to constellation points for digital modulation schemes
• Autoregressive HMMs model correlated emissions for signals with inter-symbol interference (ISI)

A practical classification architecture maintains a parallel bank of HMMs, each trained on a specific modulation hypothesis. The received signal sequence is evaluated against every model, and the modulation type corresponding to the HMM with the highest log-likelihood score is selected.

• Each HMM encodes the temporal signature of one modulation scheme
• Log-likelihood normalization prevents numerical underflow for long sequences
• Enables open-set recognition by thresholding the maximum likelihood against a null hypothesis model

Standard HMMs assume geometric state durations, which may poorly fit real channel behaviors. Hidden Semi-Markov Models (HSMMs) extend the framework by explicitly modeling state sojourn time distributions, improving classification in channels with non-exponential dwell times.

• Replaces self-transition probabilities with explicit duration distributions
• Captures long-range dependencies in fading and interference patterns
• Particularly effective for classifying frequency-hopping signals with variable hop durations

An optimal detection technique that selects the most probable transmitted symbol sequence by maximizing the likelihood function over the entire sequence. In the context of HMMs, MLSE is commonly implemented via the Viterbi algorithm, which efficiently finds the most likely sequence of hidden states (the modulation's internal memory states) given the observed signal. This is the standard method for decoding signals with inter-symbol interference.

An iterative two-step procedure for finding maximum likelihood estimates in the presence of hidden variables. For HMM-based modulation classifiers, the EM algorithm (specifically the Baum-Welch re-estimation) is the standard method for training the model when the true state sequence is unknown. It alternates between:

• E-Step: Computing the expected log-likelihood of the complete data given current parameter estimates.
• M-Step: Updating HMM parameters (transition probabilities, emission distributions) to maximize this expectation.

The natural logarithm of the likelihood function, transforming products of conditional densities into sums for numerical stability. In HMM-based classification, the log-likelihood of an observed signal sequence under each candidate modulation model is computed using the forward algorithm. The classifier then selects the modulation hypothesis that yields the highest log-likelihood score, avoiding underflow from multiplying many small probabilities.

A non-symmetric measure of how one probability distribution diverges from a reference distribution. In modulation classification, KL divergence quantifies the discriminability between HMMs representing different modulation schemes. A larger KL divergence between two models indicates that their emitted signal statistics are more distinct, leading to lower classification error probability. It provides a theoretical bound on classifier performance.

A model selection criterion that balances goodness-of-fit with model complexity by penalizing the log-likelihood with a term proportional to the logarithm of the sample size. In HMM-based classification, BIC is used to determine the appropriate number of hidden states for a modulation model. It prevents overfitting by favoring simpler HMM topologies unless additional states provide a statistically significant improvement in explaining the observed signal.

A dynamic programming procedure that computes the posterior probabilities of hidden states at each time step given the entire observation sequence. Unlike the Viterbi algorithm which finds only the single best path, the forward-backward algorithm provides soft probabilistic assignments. In modulation classification, this is used to estimate the probability that the transmitter was in a specific memory state at each symbol interval, enabling soft-decision decoding.