Representation Collapse

The embedding vectors occupy only a low-dimensional subspace of the available vector space, with many dimensions becoming zero or constant. Instead of utilizing the full representational capacity, the encoder maps all inputs to a narrow manifold. This is detectable by analyzing the singular value spectrum of the embedding matrix—a collapsed representation exhibits a rapid drop in singular values, with many near-zero entries indicating unused dimensions.

The most extreme failure mode where the encoder maps every input to an identical constant vector regardless of semantic content. The model discovers a trivial shortcut: outputting a fixed representation perfectly minimizes the invariance term of the loss but destroys all discriminative information. This is mathematically equivalent to the encoder learning a rank-0 mapping, and it commonly occurs when the variance regularization or negative pair repulsion mechanisms are insufficiently strong.

A subtler variant where embeddings retain variance but lose mutual information with the input. The vectors may appear normally distributed, yet they encode no meaningful features about the data. This arises when the encoder discards semantic content in favor of exploiting low-level statistical shortcuts—such as color histograms or patch positions—that satisfy the contrastive objective without learning transferable representations. Measuring downstream task performance reveals the collapse.

Collapse represents the extreme endpoint of the uniformity-tolerance spectrum identified in contrastive loss analysis. Perfect uniformity—where embeddings are evenly distributed on the hypersphere—maximizes information preservation but may violate semantic clustering. Conversely, excessive tolerance collapses distinct classes together. Effective training requires balancing:

• Alignment: positive pairs mapped nearby
• Uniformity: embeddings spread across the hypersphere Collapse occurs when alignment dominates and uniformity vanishes.

A precursor to collapse where the contrastive loss gradients diminish for certain feature dimensions while remaining active for others. The encoder receives no learning signal to update specific representational axes, causing them to decay toward zero through weight decay or numerical drift. Over successive training steps, these starved dimensions collapse, progressively reducing the effective rank of the embedding space until only a handful of dimensions carry all discriminative information.

The encoder discovers spurious correlations that minimize the contrastive loss without learning semantic features. Examples include:

• Chromatic aberration: matching augmented views via lens distortion patterns rather than object identity
• Patch position: using spatial location of image patches as a proxy for content
• High-frequency artifacts: relying on JPEG compression boundaries invisible to humans These shortcuts produce low training loss but catastrophic generalization failure, representing a form of collapse where representations are consistent but semantically empty.