Maximal Coding Rate Reduction (MCR2)

MCR2 learns representations by maximizing the coding rate difference between the entire dataset and the sum of individual class representations. This objective mathematically decomposes into two competing forces:

• Expansion: Maximize the volume spanned by the whole dataset to promote diversity and avoid mode collapse.
• Compression: Minimize the volume of each class's representation to collapse intra-class variation into a compact subspace. The result is a representation where each class occupies a distinct, low-dimensional linear subspace, naturally discarding instance-specific noise that inversion attacks exploit.

The compression term in MCR2 explicitly minimizes the mutual information between individual training samples and their learned representations. By forcing all instances of a class into a tight subspace, the model discards the high-frequency, instance-specific details necessary for reconstructing original inputs.

• Attackers attempting gradient inversion or feature reconstruction recover only a class-mean prototype rather than any specific training sample.
• The representation retains only what is sufficient for discrimination, implementing an information bottleneck without explicit noise injection. This provides a structural defense against model inversion that does not require differential privacy mechanisms.

MCR2 naturally induces a cosine-similarity-based nearest subspace classifier. Each class is represented by a learned linear subspace, and inference is performed by measuring the cosine similarity of a query embedding to each subspace.

• The decision boundary is determined by the principal angles between subspaces, not by Euclidean distance in the embedding space.
• This geometry makes it difficult for an adversary to interpolate between classes or generate high-confidence adversarial examples.
• The subspace structure provides a geometric interpretation of the learned features, aiding interpretability and auditing.

Unlike many deep learning objectives that rely purely on stochastic gradient descent, MCR2 admits closed-form solutions for certain architectural choices. When using a fixed encoder or in the final linear layer, the optimal representation can be derived analytically via eigendecomposition of the class-conditional covariance matrices.

• This provides theoretical guarantees on convergence and representation structure.
• The closed-form solution reveals that MCR2 is equivalent to learning a whitened and orthogonalized feature space where each class spans an independent subspace.
• This analytical tractability enables formal privacy analysis of the learned representations.

MCR2 extends naturally to both supervised and unsupervised settings:

• Supervised MCR2: Uses class labels to define the compression sets, learning maximally separable class subspaces.
• Unsupervised MCR2: Jointly learns a self-expressive affinity matrix and the representation, clustering data into subspaces without labels. This is equivalent to performing subspace clustering and representation learning simultaneously.
• Semi-supervised MCR2: Combines labeled compression with an unlabeled expansion term, leveraging abundant unlabeled data to improve the diversity of the learned representation while maintaining class structure from limited labels.

MCR2 provides a computationally tractable approximation to the Information Bottleneck (IB) principle. While IB minimizes mutual information between input and representation subject to predictive sufficiency, MCR2 operationalizes this using coding rate as a surrogate for mutual information.

• The compression term directly penalizes the rate-distortion required to encode individual samples within a class.
• Unlike variational IB methods that require training a decoder, MCR2 achieves compression purely through the geometry of the representation space.
• This connection provides theoretical justification for why MCR2 representations are naturally resistant to inversion: they explicitly minimize the information content about the input beyond what is needed for the task.

A theoretical principle that compresses input data into a latent representation that is maximally informative about the target task while minimizing mutual information with the original input. This creates a natural barrier against inversion attacks by discarding irrelevant, potentially sensitive details. MCR2 achieves a similar compression objective through a geometric, coding-theoretic lens rather than a variational one.

The process of projecting high-dimensional data into a lower-dimensional latent space. When used as a defense, it creates an information bottleneck that naturally discards the fine-grained, high-frequency details necessary for input reconstruction. MCR2 formalizes this by explicitly optimizing the coding rate of the learned subspace to be compact for each class while being expansive for the dataset as a whole.

A class of attacks that inverts intermediate feature representations or embeddings back into the original input space to extract sensitive visual or textual information. MCR2's objective of maximizing the coding rate difference forces features to be both discriminative and compressed, reducing the mutual information between the latent code and the input, thereby degrading the fidelity of any reconstructed data.

A privacy attack that reconstructs original training inputs by analyzing shared gradients in distributed learning. Representations learned via MCR2 are structured to be maximally compact per class, which reduces the variance in the gradient signals that leak input-specific information. This geometric regularization acts as a passive defense against deep leakage from gradients (DLG).

A training methodology that augments the loss function with a penalty term designed to minimize an adversary's success. MCR2 can be viewed as a form of implicit adversarial regularization where the objective naturally suppresses the encoding of instance-specific noise. By promoting subspace clustering, it hardens the model against both inversion and evasion attacks without needing a separate adversarial network.

A mathematical framework providing provable privacy guarantees by injecting calibrated noise. While MCR2 does not provide formal DP guarantees on its own, its compressed, low-rank representations are highly compatible with DP-SGD. The reduced sensitivity of the compact features means less noise is required to achieve a target privacy budget (epsilon), improving the privacy-utility trade-off.