CLIP

CLIP uses a Bi-Encoder design with two separate towers: a Vision Transformer (ViT) or ResNet for images and a Transformer for text. Each encoder independently maps its input to a fixed-dimensional embedding vector. The two encoders do not share weights, allowing the text tower to pre-compute class embeddings offline for rapid zero-shot inference. This Two-Tower Model design enables efficient dot-product scoring between image and text representations.

CLIP is trained with a symmetric InfoNCE loss over a massive batch of N image-text pairs. The objective maximizes the cosine similarity of the N correct pairs while minimizing similarity for the N² - N incorrect pairs:

• Image-to-Text: Each image must select its correct caption from the batch
• Text-to-Image: Each caption must select its correct image
• In-Batch Negatives: All other pairings within the batch serve as negative samples, creating an efficient training signal without explicit negative mining

CLIP was trained on 400 million image-text pairs collected from publicly available internet sources, known as the WebImageText (WIT) dataset. This dataset spans a vast diversity of visual concepts and natural language descriptions. Key characteristics:

• No human annotation or labeling required
• Covers everyday objects, scenes, text, logos, and abstract concepts
• The scale and diversity enable strong zero-shot generalization without task-specific fine-tuning

CLIP can classify images into arbitrary categories without any task-specific training data. The process:

1. Text Prompt Engineering: Convert class labels into natural language prompts (e.g., 'a photo of a {class}')
2. Embed Class Prompts: Encode all text prompts through the text encoder
3. Match: Compute cosine similarity between the image embedding and all text embeddings
4. Predict: Select the class with highest similarity This enables CLIP to generalize to novel visual concepts it never explicitly saw during training.

Unlike supervised models trained on fixed datasets like ImageNet, CLIP demonstrates remarkable robustness to natural distribution shifts. It achieves competitive accuracy on challenging benchmarks including:

• ObjectNet: Real-world images with unusual viewpoints and backgrounds
• ImageNet-R: Renditions, art, and cartoons of ImageNet classes
• ImageNet-Sketch: Black-and-white sketches
• ImageNet-A: Naturally adversarial examples that fool standard classifiers This robustness stems from learning from diverse, natural language supervision rather than memorizing dataset-specific features.

CLIP projects both images and text into a shared L2-normalized embedding space where semantic similarity is measured by cosine similarity. Properties of this joint space:

• Semantic Alignment: A photo of a dog is closer to the text 'a dog' than to 'a cat'
• Compositional Understanding: CLIP can bind attributes to objects (e.g., 'red cube' vs 'blue sphere')
• Cross-Modal Retrieval: Enables text-to-image and image-to-text search
• Linear Probe Quality: The learned representations transfer well to downstream tasks with simple linear classifiers