Joint Embedding Space

A joint embedding space maps heterogeneous data types into a common vector space where semantic similarity is encoded as geometric proximity. Chest X-rays and their corresponding radiology reports are projected to nearby coordinates, while unrelated pairs are pushed apart. This alignment is typically achieved through contrastive learning objectives that explicitly optimize for cross-modal distance metrics.

Once modalities share a latent space, queries in one modality can retrieve relevant items in another. Key retrieval patterns include:

• Image-to-text: Given a retinal scan, find the matching ophthalmologist's notes
• Text-to-image: Given a symptom description, retrieve relevant pathology slides
• Zero-shot classification: Classify images using only textual class descriptions without any visual training examples

Even in well-trained joint spaces, embeddings from different modalities often form distinct clusters with a measurable separation known as the modality gap. This occurs because modality-specific encoders introduce systematic biases. Mitigation strategies include:

• Adding a learnable modality offset vector to bridge the gap
• Applying contrastive objectives that explicitly penalize inter-modal distance
• Using shared encoder layers after initial modality-specific projections

In federated multi-modal fusion, joint embedding spaces must be learned without centralizing raw data. Each institution trains modality-specific encoders locally and shares only the alignment loss gradients or prototype embeddings. Federated contrastive learning ensures that cross-modal relationships learned at one hospital generalize across the network, enabling privacy-preserving cross-modal retrieval across institutional boundaries.

The dimensionality of the joint space represents a critical trade-off:

• Too low: Insufficient capacity to capture fine-grained cross-modal relationships, leading to collapsed representations
• Too high: Increased computational cost and risk of overfitting to spurious correlations Typical clinical joint spaces use 512 to 1024 dimensions, often with L2 normalization to project embeddings onto a unit hypersphere for stable cosine similarity computation.

Joint embedding quality is assessed using retrieval and alignment metrics:

• Recall@K: Fraction of queries where the correct cross-modal match appears in the top-K retrieved results
• Median rank: The median position of the correct match across all queries
• Modality alignment score: Measures the uniformity of cross-modal pair distances
• Downstream task performance: Accuracy on clinical tasks like diagnosis prediction when using the learned embeddings as input features