Embedding Pooling

Computes the arithmetic mean of all token embeddings across the sequence length, including padding tokens unless an attention mask is applied.

• Mechanism: Sums token vectors and divides by sequence length
• Best For: General-purpose semantic similarity and sentence representation
• Key Advantage: Smooths out outlier tokens and captures global context
• Masking: Apply attention masks to exclude padding tokens from the average
• Example: BERT's base uncased model uses mean pooling for sentence embeddings

Extracts only the hidden state corresponding to the special classification token prepended to every input sequence.

• Mechanism: Takes the first token's hidden state from the final layer
• Best For: Classification tasks and models pre-trained with next-sentence prediction
• Key Advantage: The CLS token is explicitly trained to aggregate sequence-level information
• Limitation: Often underperforms mean pooling for semantic similarity without task-specific fine-tuning
• Example: Original BERT paper used CLS token for sentence classification

Selects the maximum value for each dimension across all token embeddings in the sequence.

• Mechanism: Element-wise maximum over the time dimension
• Best For: Capturing salient, high-activation features when specific keywords dominate semantics
• Key Advantage: Emphasizes the most strongly activated features per dimension
• Limitation: Can discard subtle contextual information present in lower activations
• Use Case: Often applied in convolutional sentence models rather than transformer architectures

Computes a weighted sum of token embeddings where weights are learned through a self-attention mechanism, allowing the model to emphasize semantically important tokens.

• Mechanism: Learns scalar weights for each token via a softmax over attention scores
• Best For: Tasks requiring fine-grained token importance differentiation
• Key Advantage: Dynamically focuses on the most relevant tokens for the target task
• Implementation: Often uses a trainable context vector to compute attention weights
• Example: Used in some Sentence-BERT variants for improved semantic textual similarity

Uses the hidden state of the final token in the sequence, which in decoder-only architectures like GPT has attended to all preceding tokens through causal self-attention.

• Mechanism: Extracts the hidden state of the last non-padding token
• Best For: Decoder-only models where the final token aggregates full sequence context
• Key Advantage: Naturally captures sequential dependencies in autoregressive models
• Limitation: Biased toward recency; earlier context may be underrepresented
• Example: Common in GPT-style embedding extraction for semantic search

Combines multiple pooling strategies by concatenating their outputs, creating a richer representation that captures complementary aspects of the token distribution.

• Mechanism: Concatenates vectors from mean, max, and CLS pooling
• Best For: Maximizing representational power when dimensionality constraints are relaxed
• Key Advantage: Preserves both global average context and salient feature activations
• Trade-off: Increases embedding dimensionality, raising storage and compute costs
• Use Case: Research settings where accuracy is prioritized over efficiency

Utilizes the hidden state of the special [CLS] token prepended to the input sequence as the aggregate representation. This strategy relies on the model being explicitly trained during pre-training to encode sequence-level information into this single token. It is the default strategy for BERT-based models but can underperform if the model was not fine-tuned for sentence-level tasks, as the CLS token often acts as a simple "no-bias" token.

Computes the arithmetic mean of all token-level hidden states across the sequence length. This is the most common strategy for modern sentence transformers because it smooths out positional anomalies and incorporates information from all tokens equally. Attention Masking must be applied to ignore padding tokens, preventing zero-vectors from diluting the semantic signal. It generally produces more robust sentence embeddings than CLS pooling.

Selects the maximum value across the time dimension for each feature in the hidden state. This operation captures the most salient, activated features but discards the global distribution of semantics. It is highly effective when specific keywords dominate the meaning but often fails on nuanced, compositional sentences where the interaction of all words matters.

Uses learned or pre-computed attention weights to compute a weighted sum of token vectors, allowing the model to focus on semantically critical tokens while suppressing stop words. This can be implemented via a separate learned linear layer or by reusing the final layer's self-attention scores. It provides a dynamic aggregation mechanism that adapts to the specific linguistic structure of the input.

Instead of using only the final transformer layer, this strategy concatenates the hidden states from multiple layers (e.g., the last four) before pooling. This captures both high-level semantic features and lower-level syntactic nuances. While it creates a larger embedding dimension, it often yields richer representations for tasks like semantic textual similarity (STS).