Two-Tower Model

The defining characteristic of the two-tower architecture is the complete separation of the user and item encoders. Each tower processes its respective features independently:

• User Tower: Ingests user features (demographics, search history, past purchases) and outputs a normalized user embedding
• Item Tower: Ingests item features (title, description, category, price) and outputs a normalized item embedding

This independence means the item tower can pre-compute embeddings for the entire catalog offline, while the user tower generates a single vector at query time. The two sides never interact until the final dot-product scoring, enabling sub-millisecond retrieval from billion-scale corpora.

Both towers project their inputs into a shared, L2-normalized embedding space where cosine similarity reduces to a simple dot product. The relevance score between a user and item is computed as:

score(u, i) = u · i

This mathematical property is critical for efficiency:

• No cross-encoder computation at serving time
• Scores are directly compatible with Maximum Inner Product Search (MIPS) indices
• Normalization constrains embeddings to the unit hypersphere, stabilizing training and bounding scores to [-1, 1]

The shared space enforces that proximity implies semantic relevance—users are close to items they are likely to engage with.

A fundamental design constraint of two-tower models is that features are partitioned by tower:

• The user tower has access to user-side features only (historical behavior, profile attributes, real-time context)
• The item tower has access to item-side features only (metadata, content, static attributes)

This asymmetry prevents the model from learning cross-feature interactions—for example, it cannot directly model that a specific user prefers items from a particular brand when that brand is on sale. This limitation is the primary trade-off for the architecture's retrieval speed. Advanced implementations mitigate this by:

• Encoding cross-features as pre-computed user-item affinity signals in one tower
• Using the two-tower as a candidate generator followed by a cross-encoder ranker

Training a two-tower model with a full softmax over millions of items is computationally prohibitive. The standard solution is in-batch negative sampling:

• For each user in a batch of size N, the model treats the other N-1 items in the batch as negative samples
• This approximates the full softmax distribution using the random composition of the mini-batch
• The InfoNCE loss (or sampled softmax) is applied to maximize the score for the true (user, item) pair while minimizing scores for all other pairs

This technique is computationally free—no additional forward passes are needed for negatives—but introduces a popularity bias where frequently occurring items are more likely to be sampled as negatives, requiring correction mechanisms like log-Q correction or mixed negative sampling.

The separation of towers enables a critical serving optimization: offline pre-computation of the entire item catalog. The item tower runs as a batch inference job, generating embeddings for every item in the corpus:

• Item embeddings are stored in a vector database with an ANN index (typically HNSW or ScaNN)
• At serving time, only the user tower executes, producing a single query vector
• The top-k items are retrieved via approximate nearest neighbor search in logarithmic time

This decoupling means the computational cost of serving is independent of catalog size. A catalog of 100 million items requires the same inference cost as a catalog of 10,000—only the ANN search time scales (sub-linearly). Item embeddings are refreshed periodically as the catalog updates.

Two-tower models can be extended to optimize for multiple business objectives simultaneously by using multiple tower heads or weighted loss functions:

• Shared bottom layers with task-specific projection heads for click, purchase, and engagement predictions
• Each head produces a separate embedding, and final retrieval scores are a weighted combination
• Training uses a multi-task loss: L_total = α·L_click + β·L_purchase + γ·L_engagement

This allows a single model to balance immediate engagement (clicks) with long-term value (purchases). The weights α, β, γ are hyperparameters tuned to business KPIs. Advanced implementations use uncertainty weighting or gradient surgery (PCGrad) to handle conflicting gradients between objectives.

Google's seminal deployment uses a user tower trained on watch history and search queries, and a video tower trained on content features and demographics. At serving time, the user embedding is computed once and an ANN index retrieves the top-N videos from a corpus of millions, enabling sub-100ms response times for the candidate generation phase.

Major retailers deploy two-tower models to power search and browse experiences. The user tower encodes purchase history, real-time clicks, and session context. The item tower encodes product titles, descriptions, and categorical attributes. The shared embedding space enables semantic matching—a user searching for 'hiking gear' retrieves trail shoes even when the exact keyword is absent from the product title.

Apple and Google app stores use two-tower architectures where the query tower encodes the search string and the app tower encodes the app title, description, and in-app purchase behavior. The dot-product score determines relevance ranking. The dual-encoder design allows the app tower embeddings to be pre-computed and indexed nightly, while the query tower runs in real-time at sub-millisecond latency.

Social media platforms use two-tower models to retrieve relevant content from a constantly updating corpus. The user tower captures long-term interests and short-term session signals. The content tower processes article text, images, and publisher authority. The architecture naturally handles the cold-start content problem—new articles get embeddings immediately from their content features without waiting for interaction data.

Spotify and similar platforms use two-tower models where the listener tower encodes play history, skip rates, and playlist context, while the track tower encodes audio features, genre tags, and co-occurrence patterns. The shared embedding space enables cross-modal retrieval—a user who listens to acoustic folk discovers instrumental lo-fi tracks through proximity in the latent space rather than explicit genre matching.

Programmatic advertising platforms use two-tower models to match users with relevant ads in real-time bidding environments. The user tower processes browsing behavior and purchase intent signals. The ad tower encodes creative content, landing page semantics, and advertiser category. The dot-product score determines relevance before auction, filtering billions of candidates to a manageable set for downstream ranking models.