Decision Boundary

The decision boundary is the set of points equidistant between two adjacent constellation points, forming Voronoi regions. Each region is a convex polygon containing all IQ locations closer to its centroid than any other. For a minimum distance decoder operating in additive white Gaussian noise (AWGN), these boundaries are the perpendicular bisectors of the lines connecting neighboring symbols, creating an optimal tessellation that minimizes the probability of symbol error.

In uncoded, memoryless modulation with AWGN, the optimal decision boundaries are linear hyperplanes—straight lines in the IQ plane. However, in practical systems, boundaries become non-linear due to:

• Phase noise warping the constellation into arcs
• Non-Gaussian interference from co-channel signals
• Fading channels that scale and rotate the entire constellation
• Non-uniform a priori probabilities from probabilistic shaping These non-linear boundaries require more sophisticated classifiers, such as neural networks or Gaussian mixture models, to approximate the true Bayesian decision regions.

As modulation order increases, the density of decision boundaries in the IQ plane grows exponentially. A QPSK constellation has only 4 regions separated by 2 orthogonal boundaries. In contrast, a 4096-QAM constellation contains thousands of tightly packed Voronoi cells, making the boundaries extremely sensitive to small impairments. The minimum Euclidean distance between boundaries shrinks, requiring higher Error Vector Magnitude (EVM) performance and more precise synchronization to avoid crossing into an adjacent region.

Channel impairments distort the ideal decision boundaries in predictable ways:

• Carrier Frequency Offset (CFO) causes the entire boundary structure to rotate continuously, making static boundaries useless without compensation
• IQ Imbalance stretches the boundaries into an elliptical grid, altering the equidistant relationship between points
• Phase noise introduces a random angular jitter, effectively blurring the boundaries into probabilistic transition zones Blind equalization algorithms like the Constant Modulus Algorithm (CMA) work to restore the circular or square symmetry of the boundaries before symbol decisions are made.

Modern coded systems rarely use hard decision boundaries. Instead, they employ soft decision logic that outputs a likelihood metric—typically a log-likelihood ratio (LLR)—for each bit. The boundary becomes a gradient rather than a hard threshold. Points near the boundary contribute low-confidence LLRs, while points deep within a Voronoi region produce high-confidence values. This soft information is essential for modern error-correcting codes like LDPC and turbo codes, which can recover bits even when the symbol crosses the hard decision boundary.

The optimal decision boundary is derived from Bayesian decision theory, which minimizes the probability of classification error by assigning each point to the class with the maximum a posteriori probability. The boundary occurs where the posterior probabilities of two adjacent classes are equal. Under AWGN with equal priors, this reduces to the minimum Euclidean distance rule. When prior probabilities are unequal—as in probabilistic shaping—the boundary shifts toward the less likely symbol, expanding the decision region of the more probable point to reduce the overall error rate.