Joint Embedding Visualization

t-Distributed Stochastic Neighbor Embedding preserves local structure by modeling pairwise similarities in high-dimensional space and mapping them to a low-dimensional space using a Student's t-distribution. For joint embeddings, t-SNE excels at revealing fine-grained modality clusters and local neighborhoods.

• Perplexity: A hyperparameter controlling the balance between local and global structure; typical values range from 5 to 50.
• Key Insight: Points from different modalities that map to the same region indicate strong cross-modal semantic alignment.
• Limitation: Global distances are not preserved; cluster sizes and inter-cluster distances are not directly interpretable.

Uniform Manifold Approximation and Projection constructs a fuzzy topological representation of the high-dimensional data and optimizes a low-dimensional embedding to be as structurally similar as possible. UMAP is widely preferred for joint embedding visualization due to its superior preservation of global structure and faster runtime.

• n_neighbors: Controls the balance between local and global structure; higher values capture broader manifold topology.
• min_dist: The minimum distance between embedded points, controlling clustering tightness.
• Advantage: Better preserves the continuous manifold structure of shared semantic spaces compared to t-SNE.

Principal Component Analysis performs an orthogonal linear transformation to project data onto the directions of maximum variance. While PCA is often used as an initialization step for t-SNE or UMAP, it also serves as a baseline diagnostic for joint embedding spaces.

• Explained Variance Ratio: Quantifies how much of the total cross-modal variance is captured by each principal component.
• Use Case: Rapidly assess whether modalities are linearly separable or if nonlinear methods are required.
• Limitation: Cannot capture nonlinear manifold structures common in deep multimodal representations.

Pairwise Controlled Manifold Approximation is a recent method designed to preserve both local and global structure without the parameter sensitivity of t-SNE. PaCMAP uses three distinct loss terms to balance neighbor preservation, mid-near pair attraction, and far pair repulsion.

• Key Innovation: Explicitly optimizes for mid-range distances, preventing the fragmentation of global structure seen in t-SNE.
• Joint Embedding Utility: Effectively reveals the continuous semantic gradients between modalities without distorting the overall manifold shape.
• Stability: Less sensitive to hyperparameter choices than t-SNE or UMAP.

Visual inspection must be complemented by quantitative metrics that measure the trustworthiness and fidelity of the low-dimensional projection.

• Trustworthiness: Measures the proportion of points that are neighbors in the low-dimensional space that are also neighbors in the high-dimensional space.
• Continuity: Measures the proportion of high-dimensional neighbors preserved in the low-dimensional embedding.
• Neighborhood Preservation Ratio: The fraction of k-nearest neighbors retained after projection.
• Cross-Modal Alignment Score: A custom metric measuring the mixing ratio of different modalities within local neighborhoods, where high mixing indicates strong semantic alignment.

Quantitative comparison of internal representations from different modalities using metrics like Centered Kernel Alignment (CKA) or Procrustes distance. This analysis reveals how well a model aligns text and image embeddings in the joint space before fusion. Engineers use these similarity scores to diagnose modality gaps—systematic offsets between embedding distributions that indicate failed alignment. High similarity suggests the model has learned a shared semantic structure; low similarity points to training deficiencies or modality-specific biases that require architectural intervention.

A diagnostic metric measuring the uncertainty in how a model distributes attention across modalities at the fusion point. Low entropy indicates the model heavily favors one modality, potentially ignoring critical cross-modal signals. High entropy suggests balanced integration. Calculated from the softmax distribution over modality-specific attention weights, this metric helps engineers detect modality bias—for instance, a vision-language model that defaults to text when images are ambiguous. Tracking fusion entropy across layers reveals where integration actually occurs in the architecture.

A method for tracking the propagation of attention weights across modalities through successive transformer layers. By aggregating attention matrices, engineers can visualize how information from image patches flows to text tokens and vice versa. The technique produces flow graphs showing which layers perform early fusion versus late integration. This is critical for debugging vision-language models where a prediction should depend on specific cross-modal interactions—for example, verifying that a caption truly attends to the described object rather than relying on dataset priors.

A technique from causal inference applied to identify specific cross-modal neurons that function as mediators between input concepts and outputs. Engineers intervene on activations during forward passes—zeroing out or swapping representations—to measure the causal effect on predictions. For example, mediating the 'color' representation from an image while keeping text unchanged isolates whether the model uses visual color information. This goes beyond correlation-based attribution to establish necessary and sufficient conditions for cross-modal reasoning.

An architectural intervention that forces a model to predict human-interpretable concepts spanning modalities before making final predictions. For instance, a model must first predict 'contains dog,' 'outdoor scene,' and 'action: running' from joint embeddings before classifying an image-text pair. This enables direct inspection of the cross-modal reasoning chain. Engineers can audit which concepts activated and why, transforming opaque joint embeddings into auditable decision pathways. The trade-off is potential accuracy loss for full interpretability.

A specific diagnostic quantifying the focus of cross-modal attention heads. Low entropy indicates sharp, specific grounding—a text token attending to a precise image region. High entropy signals broad, unfocused attention across the entire image, suggesting the model hasn't learned meaningful alignment. Engineers plot entropy distributions per attention head to identify specialized grounding heads versus redundant heads that can be pruned. This metric is essential for optimizing vision-language models where computational efficiency requires eliminating heads that don't contribute precise cross-modal mappings.