Variational Information Bottleneck

VIB formalizes the balance between compression and predictive power using the information bottleneck Lagrangian: min I(X; Z) - β * I(Z; Y). The mutual information I(X; Z) acts as a rate term penalizing complexity, while I(Z; Y) is the distortion term rewarding task relevance. The Lagrange multiplier β controls the trade-off, allowing engineers to traverse the entire rate-distortion curve for a given task.

Unlike deterministic autoencoders, the VIB encoder outputs the parameters of a conditional distribution p(z|x), typically a multivariate Gaussian. This stochasticity acts as a regularizer, forcing the representation to be robust to noise and preventing the memorization of irrelevant input details. In practice, the encoder outputs a mean vector μ and a diagonal covariance matrix Σ, from which the latent code z is sampled using the reparameterization trick.

Computing exact mutual information in high-dimensional spaces is intractable. VIB uses a variational lower bound on I(Z; Y) by approximating the true decoder p(y|z) with a neural network q(y|z). Simultaneously, the marginal p(z) is approximated by a fixed prior, typically a standard Gaussian N(0, I), allowing the KL divergence D_KL(p(z|x) || p(z)) to upper-bound the compression term I(X; Z). This dual approximation makes the objective fully differentiable.

VIB is the theoretical backbone of semantic communication systems. Instead of minimizing bit error rate, a VIB-based joint source-channel encoder learns to transmit only the information necessary for a downstream task, such as image classification at the receiver. This results in significant bandwidth savings by discarding pixel-level noise and semantically irrelevant features, achieving goal-effective rather than bit-exact transmission.

By minimizing I(X; Z), the VIB objective naturally learns representations that are maximally compressed about the input. This forces the encoder to discard any information that does not help predict the target Y. Consequently, the learned latent space becomes invariant to nuisance factors like background clutter, sensor noise, or channel variations, providing a principled form of domain adaptation without adversarial training.

VIB generalizes classical rate-distortion theory to the case where the distortion measure is the log-loss of a task classifier. The optimal achievable rate for a given task accuracy defines the information bottleneck bound. This provides a theoretical limit for lossy compression in learned communication systems, guiding the design of neural encoders that operate near the optimal rate-distortion frontier for specific inference tasks.

A neural network trained to estimate the mutual information between high-dimensional random variables, such as channel input and output. The VIB objective relies on computing mutual information, which is often intractable. MINE provides a differentiable, scalable method to maximize or minimize this quantity, serving as a critical optimization component for deep learning-based rate-distortion trade-offs.

A paradigm shift from bit-exact transmission to goal-effective communication. Instead of reconstructing the source perfectly, a joint source-channel encoder transmits only the semantic information relevant to a specific inference task at the receiver. VIB is the mathematical framework that defines this goal, learning a stochastic encoding that discards task-irrelevant noise while preserving actionable features.

A technique using a single neural network to map raw source data directly to channel symbols. Deep JSCC bypasses separate source and channel coding stages. When optimized with a VIB objective, the encoder learns a latent representation that is both bandwidth-efficient and robust to channel noise, directly maximizing task performance rather than minimizing bit error rate.

A mathematical or neural surrogate model of a physical communication channel that allows gradients to backpropagate from the receiver loss to the transmitter parameters. For VIB optimization, the channel must be differentiable to train the stochastic encoder. This model enables the joint learning of compression and robust transmission without requiring a closed-form analytical channel description.

An optimization technique that assigns a non-uniform probability distribution to constellation points to approach the Shannon capacity limit. VIB naturally learns a probabilistic shaping scheme by outputting the parameters of a distribution over the channel input space. This contrasts with fixed geometric shaping, allowing the encoder to adapt symbol probabilities dynamically to the channel state and task.

An end-to-end learned transmitter-receiver pair optimized to maximize mutual information with a legitimate receiver while minimizing leakage to an eavesdropper. The VIB objective is directly applicable here: the encoder learns a compressed representation that is maximally informative about the message for the intended recipient but forms a bottleneck that limits an adversary's ability to decode the signal.