Cross-Attention

Cross-attention establishes a directed, asymmetric relationship between two distinct sequences. One sequence provides the query vectors, while the other provides the key and value vectors. This allows the query sequence to selectively retrieve and integrate information from the key/value sequence. For example, in an image captioning model, the text decoder's hidden states (queries) attend to the encoded image features (keys/values) to generate descriptive words.

The mechanism computes a set of attention weights for each query by taking the dot product of the query with all keys, followed by a softmax. These weights act as a dynamic, content-based gate:

• High weight: The query strongly attends to that specific key/value pair.
• Low weight: The information from that key/value pair is largely ignored for the current query. This creates a soft, differentiable selection mechanism, unlike hard-coded rules or fixed gates.

Cross-attention is the primary architectural component for fusing information from different data types (modalities). It is the core of Vision-Language Models (VLMs) and Multimodal Transformers.

• Text-to-Image: Language queries attend to visual patches for visual question answering.
• Audio-to-Text: Audio frame queries attend to transcribed text for speaker diarization.
• Sensor Fusion: LiDAR point cloud queries attend to camera image features for autonomous vehicle perception.

Cross-attention layers can be inserted at different points in a model architecture, leading to distinct fusion strategies:

• Early Fusion: Cross-attention is applied to raw or lightly processed features from each modality, enabling deep, intertwined processing (e.g., Perceiver IO).
• Late Fusion: Each modality is processed independently by separate encoders, and cross-attention is used only at a high level to combine final representations for a task.
• Hierarchical Fusion: Multiple cross-attention layers are used at different depths of the network, allowing progressive refinement of cross-modal understanding.

It is critical to distinguish cross-attention from the more common self-attention:

• Self-Attention: Queries, keys, and values are all derived from the same sequence. It models intra-sequence relationships (e.g., word-to-word dependencies in a sentence).
• Cross-Attention: Queries come from one sequence, while keys and values come from a different sequence. It models inter-sequence or inter-modal relationships. A Transformer decoder block typically contains one layer of masked self-attention (for the target sequence) followed by one layer of cross-attention (where the target attends to the encoder output).

The computational cost of cross-attention is a key design constraint. The complexity is O(n * m), where n is the length of the query sequence and m is the length of the key/value sequence.

• This can become expensive when fusing long sequences (e.g., high-resolution images or long audio clips).
• Common optimization strategies include:
  • Pooling or downsampling the key/value sequence.
  • Using efficient attention approximations (e.g., linear attention, pooling attention).
  • Employing hierarchical cross-attention where high-level summaries are attended to first.