Information Bottleneck

The Information Bottleneck objective formalizes the trade-off between compression and prediction by minimizing the mutual information I(X;Z) between the input X and the latent representation Z, while maximizing I(Z;Y) with the target Y. This Lagrangian formulation, min I(X;Z) - β I(Z;Y), creates a bottleneck that forces the encoder to discard irrelevant features. The hyperparameter β controls the compression-accuracy trade-off: higher β values prioritize task performance, while lower β values enforce stronger compression and greater privacy. This principled information-theoretic approach provides a tunable mechanism for limiting the amount of raw input detail preserved in latent codes.

By explicitly minimizing I(X;Z), the Information Bottleneck objective directly counteracts the information leakage exploited by model inversion attacks. When the latent representation Z contains minimal mutual information with the input X, an adversary attempting to reconstruct X from Z faces a fundamental information-theoretic barrier. Unlike ad-hoc defenses such as dropout or dimensionality reduction, the IB framework provides provable bounds on the maximum fidelity of any reconstruction attempt. This makes IB-trained models inherently resistant to both white-box gradient inversion and black-box feature reconstruction attacks, as the necessary high-frequency details are systematically discarded during encoding.

The Variational Information Bottleneck provides a practical, deep learning-compatible approximation of the theoretical IB objective. VIB replaces the intractable mutual information terms with variational bounds:

• A parametric encoder q(z|x) approximates the optimal stochastic mapping
• A variational marginal r(z) approximates the latent prior, typically a standard Gaussian
• The KL divergence D_KL(q(z|x) || r(z)) upper-bounds I(X;Z) This formulation integrates seamlessly with stochastic gradient descent and can be implemented as a modified loss function with a β-weighted KL regularization term, making IB principles accessible for large-scale neural network training.

The Information Bottleneck framework distinguishes between two encoding paradigms with different privacy implications:

• Deterministic bottlenecks: The encoder maps each input to a single latent point. While computationally simpler, deterministic mappings can preserve invertible structure that aids reconstruction attacks.
• Stochastic bottlenecks: The encoder outputs a distribution over latent codes, typically a Gaussian with learned mean and variance. The reparameterization trick enables gradient flow through the sampling operation. Stochastic encoding provides stronger privacy guarantees by introducing uncertainty that compounds with the compression loss, making exact inversion statistically impossible even with unlimited queries. The stochastic variant is essential for achieving meaningful I(X;Z) minimization in practice.

The Information Bottleneck can be understood through the lens of rate-distortion theory, where:

• The rate corresponds to I(X;Z), measuring how many bits of input information are preserved in the representation
• The distortion corresponds to the prediction error on task Y, measuring the cost of compression
• The IB curve traces the optimal achievable rate-distortion pairs, defining the Pareto frontier of compression versus accuracy This perspective reveals that IB-based defenses do not simply degrade model utility; they identify the most efficient representations that achieve a target task performance while discarding the maximum amount of extraneous input information. Privacy gains come from eliminating redundancy, not from adding noise.

While distinct, the Information Bottleneck and Differential Privacy (DP) are complementary defense mechanisms:

• IB provides an information-theoretic bound on what any adversary can extract from representations, regardless of computational power
• DP provides a worst-case statistical guarantee that individual training samples cannot be distinguished from the model's outputs
• Combining IB compression with DP-SGD creates a layered defense: IB limits the information content of latent codes, while DP-SGD bounds the influence of individual training examples on the model parameters This synergy is particularly powerful in federated learning, where IB-compressed updates reduce the attack surface for gradient leakage, and DP noise provides formal membership inference protection.