Mutual Information Neural Estimator

MINE formulates mutual information estimation as a dual optimization problem using the Donsker-Varadhan representation. A statistics network (T) is trained to maximize a lower bound on MI, while the main transceiver network is optimized to maximize the estimated MI. This adversarial-like framework allows gradient-based training without explicit probability density estimation.

Unlike traditional non-parametric estimators (e.g., k-NN or kernel density), MINE scales to high-dimensional continuous random variables such as raw I/Q sample blocks or learned latent representations. The neural network effectively learns a low-dimensional sufficient statistic, making it practical for modern communication systems with large constellations and multiple antennas.

MINE provides a fully differentiable estimate of mutual information that can be used as a loss function. This allows end-to-end autoencoders to directly maximize I(X;Y) between channel input X and output Y, optimizing spectral efficiency without relying on proxy metrics like bit error rate or mean squared error.

The MINE estimator is strongly consistent, meaning it converges to the true mutual information as sample size increases. By using exponential moving averages of partition function estimates during training, MINE reduces bias that would otherwise lead to overestimation, a critical property for reliable capacity approximation.

MINE operates directly on paired samples from the joint distribution P(X,Y) and unpaired samples from the marginal product P(X)P(Y). This sample-based approach eliminates the need for explicit channel models during training, enabling optimization over real measured channel data or complex stochastic simulators.

MINE serves as a foundational component in variational information bottleneck architectures for task-oriented communication. By estimating the mutual information between a learned representation and both the input source and the downstream task, MINE enables precise control of the compression-relevance trade-off in semantic communication systems.