Contrastive Language-Image Pre-training (CLIP)

CLIP employs two separate neural network towers—an image encoder and a text encoder—that independently map their respective inputs into a shared, high-dimensional embedding space.

• Image Encoder: Typically a Vision Transformer (ViT) or a ResNet that processes raw pixels into a fixed-length vector.
• Text Encoder: A Transformer-based model that converts a natural language description into a vector of identical dimensionality.
• Shared Projection: Both encoders project their outputs through a linear layer into a joint multi-modal embedding space where cosine similarity is computed.

This decoupled design allows for efficient batch processing of image-text pairs without cross-modal attention during encoding.

The model is trained using a symmetric cross-entropy loss over a batch of N matched image-text pairs, creating an N×N similarity matrix.

• Positive Pairs: The N diagonal elements represent correctly matched image-text pairs that the model maximizes similarity for.
• Negative Pairs: The N²-N off-diagonal elements are implicitly treated as negatives, pushed toward zero similarity.
• Temperature Parameter: A learned scalar τ scales the logits, controlling the sharpness of the softmax distribution and the concentration of the similarity scores.

This InfoNCE loss formulation effectively trains on N² supervision signals per batch, dramatically increasing sample efficiency compared to predictive objectives.

CLIP's joint embedding space enables classification without any task-specific training data by framing it as a retrieval problem.

• Prompt Engineering: Class labels are embedded in natural language templates, e.g., "a photo of a {class}" to provide contextual framing.
• Ensembling Prompts: Multiple prompt templates are averaged at the embedding level to improve robustness, such as "a centered satellite photo of {class}" or "a black and white photo of the {class}".
• Caching Text Features: The text encoder can pre-compute embeddings for all candidate classes once, making inference a simple nearest-neighbor lookup against the image embedding.

This mechanism allows CLIP to generalize to entirely novel visual concepts described in natural language without retraining.

CLIP was trained on 400 million image-text pairs collected from publicly available internet sources, a dataset known as WIT (WebImageText).

• Natural Language Supervision: Unlike curated datasets like ImageNet with fixed class labels, CLIP learns from noisy, descriptive captions that contain rich semantic information.
• Scale Over Cleanliness: The training paradigm prioritizes dataset scale and diversity over meticulous human annotation, relying on the contrastive objective to filter noise.
• Distributional Breadth: The corpus spans a vast range of visual concepts, enabling the model to develop a broad understanding of everyday objects, scenes, and abstract concepts.

This massive, diverse dataset is the primary driver behind CLIP's robust zero-shot capabilities.

CLIP's zero-shot performance can be significantly improved by applying a logit bias to correct for the model's inherent prior over class labels.

• Class Imbalance: The model may have seen certain concepts more frequently during pre-training, leading to an unbalanced prior distribution.
• Calibration Step: A small set of unlabeled images from the target distribution is used to estimate the marginal distribution of the model's predictions.
• Bias Correction: The estimated prior log-probabilities are subtracted from the raw logits, calibrating the model to a uniform prior over the target classes.

This simple post-processing step often yields a substantial accuracy boost on downstream benchmarks.

While early CLIP models used modified ResNets, the ViT-L/14 variant demonstrated superior performance and has become the standard backbone.

• Patch Embedding: Input images are divided into non-overlapping 14×14 pixel patches, linearly projected into the transformer's hidden dimension.
• Global Self-Attention: Unlike CNNs with local receptive fields, the ViT applies self-attention across all patches from the first layer, capturing long-range dependencies.
• Joint Training: The ViT is trained from scratch alongside the text transformer using the contrastive objective, not pre-trained on ImageNet, resulting in visual features optimized for semantic alignment with language.

This integration yields a unified architecture that excels at both fine-grained recognition and holistic scene understanding.

A shared latent vector space where representations of different data modalities—such as medical images and clinical text—are mapped to enable direct comparison and cross-modal retrieval. In CLIP, this space is learned by maximizing the cosine similarity of matched image-text pairs while minimizing it for mismatched pairs. Key properties:

• Enables zero-shot transfer by comparing image embeddings to text embeddings of class labels
• Dimensionality typically ranges from 512 to 1024
• Distance metrics like cosine similarity define semantic relatedness

An attention mechanism where the representation of one modality is used to guide the feature extraction or focus of another. While CLIP uses a dual-encoder architecture without cross-attention between modalities, this concept is critical in alternative fusion architectures. In federated multi-modal fusion:

• Allows radiology reports to contextually inform image feature extraction
• Enables one data stream to query relevant information from another
• Contrasts with CLIP's late interaction approach where modalities are encoded independently

Independent neural network branches designed to extract salient features from a single data type before passing the resulting embeddings to a shared fusion module. CLIP employs a dual-encoder design:

• Image Encoder: Typically a Vision Transformer (ViT) or ResNet that produces a fixed-dimensional embedding
• Text Encoder: A transformer-based language model that generates a corresponding text embedding
• Both encoders are trained jointly via contrastive loss to align their output spaces
• In federated settings, these encoders can be trained on separate clients with different modality availability

A self-supervised federated framework where local models are trained to pull representations of augmented views of the same sample together while pushing apart representations of different samples. Relationship to CLIP:

• Extends the contrastive paradigm from cross-modal pairs to intra-modal augmentations
• Local clients can pre-train encoders on unlabeled data before federated aggregation
• Reduces reliance on labeled data across distributed sites
• Can be combined with CLIP-style objectives for multi-modal federated pre-training
• Key benefit: Learns robust representations without sharing raw patient data

The task of searching for data in one modality using a query from another. CLIP's joint embedding space makes this possible without task-specific training. Clinical applications:

• Retrieving relevant radiology reports using a chest X-ray as the search input
• Finding similar medical images based on a textual description of findings
• Enabling semantic search across heterogeneous hospital data silos
• In federated settings, retrieval can occur across institutional boundaries without centralizing data
• Zero-shot capability: No fine-tuning needed for new retrieval tasks

A decentralized training paradigm where clients learn useful representations from unlabeled local data using pretext tasks. CLIP's contrastive objective is inherently self-supervised, using naturally occurring image-text pairs as supervision. In federated healthcare:

• Hospitals can leverage unlabeled PACS images and associated reports
• No manual annotation required at any site
• Local encoders learn clinically meaningful features before aggregation
• Reduces the annotation bottleneck that plagues medical AI
• Pretext task: Predicting which report matches which image within a batch