Token-Level Interaction

Unlike Bi-Encoders that produce a single fixed vector, the Cross-Encoder computes an attention weight for every query token against every document token. This creates a full n × m interaction matrix where:

• The model learns which specific query terms are most relevant to which document terms
• Exact match signals (e.g., "Python" matching "Python") receive high attention weights
• Semantic alignments (e.g., "automobile" aligning with "car") are captured through contextualized embeddings
• The [CLS] token aggregates these pairwise interactions into a single relevance score

Token-level interaction inherently preserves lexical precision that dense vector similarity often loses. When a query contains a rare proper noun or technical identifier:

• The Cross-Encoder's attention heads can directly match identical tokens across sequences
• This prevents the vocabulary mismatch problem where semantically similar but lexically different terms fail to retrieve relevant documents
• Critical for enterprise search where product codes, error messages, and legal citations require exact matching
• The model learns to balance semantic understanding with literal term overlap through multi-head attention

Self-attention enables dynamic term importance based on the full query-document context rather than static IDF weights:

• A common word like "bank" receives different attention depending on whether the document discusses finance or rivers
• Negation terms ("not," "never") can flip relevance through attention to surrounding tokens
• Cross-attention heads specialize in different linguistic phenomena: one head may focus on entity matching while another handles predicate-argument alignment
• This eliminates the need for hand-crafted feature engineering used in traditional Learning to Rank systems

The transformer's positional encodings combined with token-level interaction allow the model to assess structural alignment between query and document:

• Proximity of matched terms is implicitly scored through attention patterns
• The model can detect when query terms appear in the same sentence versus scattered across paragraphs
• Phrase-level matching emerges naturally: "machine learning" in the query attends strongly to contiguous "machine learning" in the document
• This positional awareness distinguishes Cross-Encoders from bag-of-words lexical models like BM25

The expressiveness of token-level interaction comes with a quadratic complexity cost that defines the Cross-Encoder's role in search architectures:

• Computing the full attention matrix scales as O(n²) with sequence length, making real-time scoring over millions of documents infeasible
• This is why Cross-Encoders are deployed as re-rankers on a pre-filtered candidate set (typically top-100 or top-1000)
• A typical query-document pair of 512 tokens requires approximately 262,144 attention computations
• Production systems mitigate this through INT8 quantization, ONNX Runtime optimizations, and dynamic batching to achieve sub-100ms latency per pair

Token-level interaction in Cross-Encoders differs fundamentally from late interaction models like ColBERT:

• Cross-Encoder: All query tokens attend to all document tokens simultaneously during encoding, producing a single holistic relevance representation
• ColBERT/MaxSim: Query and document are encoded independently, and interaction is delayed to a final cosine similarity computation between token embeddings
• Cross-Encoders capture richer dependencies (e.g., how query term A modifies the interpretation of query term B when both attend to document term C)
• Late interaction sacrifices this depth for the ability to pre-compute document token embeddings, enabling faster retrieval at scale