Biaffine Attention

Unlike a standard bilinear transformation that requires a full weight tensor, biaffine attention uses a low-rank factorization to reduce parameters. It computes the score between a head vector h and a dependent vector d as: s = h^T U d + W[h; d] + b. The U matrix captures multiplicative interactions, while the W term adds a linear component. This factorization prevents overfitting and dramatically reduces memory consumption when the number of possible relations is large.

The mechanism is called 'deep' biaffine when separate multi-layer perceptrons (MLPs) transform the input representations before the bilinear scoring. This creates a task-specific subspace for the head and dependent roles:

• Head MLP: Projects predicate representations into a lower-dimensional space optimized for governing relations
• Dependent MLP: Projects argument candidate representations into a space optimized for receiving relations
• Deep biaffine consistently outperforms shallow variants by learning non-linear interactions before the final affinity computation

Biaffine attention scores each predicate-argument pair independently of other arguments. This design choice enables:

• Parallel computation: All candidate spans can be scored simultaneously against a predicate
• Global normalization: A softmax over all candidates produces a valid probability distribution
• Simplicity: No complex structured prediction or beam search is required during inference The independence assumption works well because the deep MLP encoders already capture contextual dependencies from the surrounding sentence.

In span-based SRL, biaffine attention scores arbitrary text spans rather than single tokens. A span from position i to j is represented by concatenating:

• The endpoint embeddings of the span boundaries from a BiLSTM or Transformer
• A span-width feature embedding to encode the length of the candidate argument
• An attention-weighted sum of all token representations within the span This representation is then fed through the dependent MLP before biaffine scoring against the predicate.

For multi-class argument classification, the biaffine mechanism extends to a third-order tensor. Instead of a single score per pair, it produces a vector of scores—one for each semantic role label:

• The U matrix becomes a U tensor with dimensions [d_head, d_dep, num_labels]
• Each slice of the tensor corresponds to a specific role like ARG0, ARG1, or ARGM-LOC
• A softmax over the label dimension yields the probability distribution over possible semantic roles This allows the model to jointly identify and classify arguments in a single forward pass.

Modern implementations replace the original BiLSTM encoders with BERT or RoBERTa representations. The workflow becomes:

1. Feed the sentence through a pre-trained Transformer to get contextualized token embeddings
2. Apply the deep biaffine attention head directly on top of these embeddings
3. Fine-tune the entire model end-to-end This combination achieves state-of-the-art results on the CoNLL-2012 and OntoNotes 5.0 benchmarks, with F1 scores exceeding 87% for span-based SRL.

The mathematical core of biaffine attention. Instead of a full weight tensor, it factorizes the transformation into two smaller matrices, drastically reducing parameters while preserving expressiveness.

• Mechanism: Computes score = h_arg^T * U * h_pred where U is a low-rank matrix.
• Benefit: Prevents overfitting on small treebanks by limiting model capacity.
• Origin: Introduced by Dozat & Manning (2017) for dependency parsing.

A specific instantiation of biaffine attention used for argument classification in SRL. It separately scores each candidate span against the predicate representation.

• Input: Contextualized embeddings for the predicate and a candidate argument span.
• Output: A probability distribution over semantic role labels (e.g., ARG0, ARG1, ARGM-LOC).
• Advantage: Decouples the role label from the syntactic distance, allowing the model to learn label-specific affinities.

The original application of biaffine attention. It scores the likelihood of a directed edge between a head word and a dependent word in a syntactic parse tree.

• Head Score: Computes the probability that word i is the syntactic head of word j.
• Label Score: A separate biaffine classifier predicts the specific dependency relation (e.g., nsubj, dobj).
• Result: Enables globally optimal parsing via the Chu-Liu/Edmonds algorithm.

A critical preprocessing step for span-based SRL models using biaffine attention. Raw token embeddings within a candidate argument span must be aggregated into a fixed-size vector.

• Max Pooling: Takes the element-wise maximum across token embeddings.
• Mean Pooling: Averages token embeddings.
• Endpoint Concatenation: Concatenates only the first and last token embeddings, often combined with a learned span-width feature.

To make biaffine scoring computationally feasible, the number of candidate spans must be drastically reduced before the attention calculation.

• Constituent Pruning: Uses a syntactic parser to propose only valid phrase-structure constituents.
• Heuristic Pruning: Filters spans by maximum width or total count.
• Learned Pruning: A lightweight feedforward network scores spans for likelihood of being any argument, keeping only the top-k candidates.

Biaffine attention excels in architectures that jointly model syntactic parsing and semantic role labeling. Shared representations improve generalization.

• Shared Encoder: A single BiLSTM or Transformer processes the input text.
• Syntax Head: A biaffine arc scorer predicts the dependency tree.
• SRL Head: A biaffine classifier predicts semantic roles, using the syntactic head's hidden states as a structural prior.
• Benefit: Syntax acts as a regularizer, guiding the SRL model toward linguistically plausible argument boundaries.