Glossary

Biaffine Attention

A scoring mechanism used in mention-ranking models that computes a pairwise score between mention and antecedent representations using a learned bilinear transformation.

Get in touch Learn more

ML engineer managing model training cluster on laptop, GPU utilization visible, technical deep learning setup.

PAIRWISE SCORING MECHANISM

What is Biaffine Attention?

A neural scoring function that computes a pairwise compatibility score between two vector representations using a learned bilinear transformation, widely used in dependency parsing and coreference resolution.

Biaffine attention is a scoring mechanism that computes a pairwise score between two input vectors—typically a mention representation and a candidate antecedent representation—using a learned bilinear transformation. Unlike simple dot-product attention, it applies a learned weight matrix U to capture asymmetric interactions: score(x, y) = x^T U y + W[x; y] + b. This allows the model to learn distinct roles for the two inputs, making it ideal for tasks where the relationship is directional, such as determining if a pronoun refers to a specific noun phrase.

In coreference resolution, biaffine attention powers the mention-ranking architecture by scoring every candidate antecedent for a given mention span. The model computes a biaffine score between the mention's span representation and each antecedent's span representation, then selects the highest-scoring candidate. This mechanism naturally handles the asymmetry of the coreference decision—the model can learn that a pronoun seeking an antecedent behaves differently than a proper noun being referenced—and enables efficient, parallelized computation over all candidate pairs.

MECHANISM

Key Characteristics of Biaffine Attention

Biaffine attention is a learned scoring function that computes pairwise compatibility between two vector representations using a bilinear transformation, forming the core of modern mention-ranking coreference architectures.

Bilinear Scoring Mechanism

Unlike simple dot-product attention, biaffine attention uses a learned weight matrix to capture complex interactions between mention and antecedent representations. The score is computed as:

score(m, a) = m^T W a + U[m; a] + b
The bilinear term m^T W a models multiplicative feature interactions
The affine term U[m; a] captures additive interactions
This parameterization allows the model to learn asymmetric relationships between spans

Pairwise Antecedent Scoring

Biaffine attention computes an independent score for every candidate antecedent-mention pair in the document. The architecture:

Takes a mention representation and an antecedent representation as input
Projects both through separate feedforward networks before the biaffine layer
Produces a scalar compatibility score for each pair
Enables the mention-ranking model to select the highest-scoring antecedent from all candidates

Deep Biaffine Architecture

The canonical formulation from Dozat & Manning (2017) introduced deep biaffine attention for dependency parsing, later adapted to coreference. Key architectural properties:

Uses stacked MLPs to transform input representations before the biaffine operation
Applies dropout to the transformed representations for regularization
The biaffine weight tensor has dimensions d x c x d where d is hidden size and c is number of classes
For coreference, c=1 produces a single scalar score indicating coreference likelihood

Integration in Mention-Ranking Models

In end-to-end neural coreference systems, biaffine attention replaces simpler scoring functions to improve accuracy:

Span representations from SpanBERT or similar encoders serve as input
A feedforward network projects each span into a task-specific space
Biaffine scoring computes pairwise compatibility between the current mention and all candidate antecedents
The model applies a softmax over antecedents to select the most likely coreferent link
Higher-order inference iteratively refines span representations using predicted antecedents

Advantages Over Feedforward Scoring

Biaffine attention offers several benefits compared to simple concatenation-based scoring:

Multiplicative interactions: Captures feature conjunctions that additive models miss
Parameter efficiency: The bilinear form uses fewer parameters than a large feedforward network over concatenated inputs
Asymmetric relationships: The weight matrix W can learn that certain mention types prefer specific antecedent types
Improved generalization: The structured parameterization reduces overfitting on small coreference datasets like CoNLL-2012

Relationship to Standard Attention

Biaffine attention generalizes standard attention mechanisms used in transformers:

Dot-product attention: score(q, k) = q^T k is a special case where W = I and the affine term is zero
Additive attention: Uses a feedforward network over concatenated vectors, equivalent to the affine term alone
Biaffine attention: Combines both, learning a full interaction matrix between the two representations
This makes biaffine attention strictly more expressive while remaining computationally efficient for the smaller candidate sets in coreference

BIAFFINE ATTENTION

Frequently Asked Questions

Explore the mechanics of biaffine attention, a powerful scoring mechanism that computes pairwise relationships between mentions and antecedents in neural coreference resolution systems.

Biaffine attention is a pairwise scoring mechanism that computes a compatibility score between two vector representations using a learned bilinear transformation. Unlike standard dot-product attention, which uses a simple scalar product, biaffine attention introduces a learned weight matrix between the two vectors. The mechanism takes two input representations—typically a mention span representation and a candidate antecedent representation—and passes them through separate feedforward networks before applying the bilinear form: score(i,j) = x_i^T W y_j + U[x_i; y_j] + b. This formulation captures both the multiplicative interaction between the two vectors via the weight matrix W and an additive bias term that considers each vector independently. In coreference resolution, this allows the model to learn complex compatibility patterns between mentions and their potential antecedents, such as gender agreement, number agreement, and semantic compatibility, all within a single, differentiable scoring function.

SCORING FUNCTION COMPARISON

Biaffine Attention vs. Other Scoring Mechanisms

Comparison of pairwise scoring functions used in mention-ranking coreference models to compute compatibility between a mention and a candidate antecedent.

Feature	Biaffine Attention	Feedforward MLP	Cosine Similarity
Scoring function	Bilinear transformation with learned weight matrix	Concatenation followed by nonlinear projection	Dot product of normalized vectors
Parameter count	d × d + d	2d × h + h × 1	0
Asymmetric scoring
Captures feature interactions
Computational complexity	O(d²)	O(d × h)	O(d)
Trainable parameters
Used in coreference models
Requires same-dimensional inputs

Enabling Efficiency, Speed & Accuracy

Intelligent Analysis, Decision & Execution

We build AI systems for teams that need search across company data, workflow automation across tools, or AI features inside products and internal software.

Talk to Us

Search across company data

Give teams answers from docs, tickets, runbooks, and product data with sources and permissions.

Useful when people spend too long searching or get different answers from different systems.

Enterprise searchRAGPermissions

Automate internal workflows

Use AI to route work, draft outputs, trigger actions, and keep approvals and logs in place.

Useful when repetitive work moves across multiple tools and teams.

AI agentsWorkflow automationGovernance

Add AI to products and internal tools

Build assistants, guided actions, or decision support into the software your team or customers already use.

Useful when AI needs to be part of the product, not a separate tool.

AI integrationDecision supportModel routing

COREFERENCE ARCHITECTURE

Related Terms

Explore the core concepts surrounding Biaffine Attention, a foundational scoring mechanism for high-precision mention-ranking models in neural coreference resolution.

Mention-Ranking Model

The primary architecture that utilizes biaffine attention to score candidate antecedents. Instead of making independent pairwise decisions, it evaluates all potential antecedents for a given mention and selects the highest-scoring one. This approach naturally handles the non-local dependencies common in discourse by learning a global ranking function over the candidate set.

Mention Pair Model

A precursor to mention-ranking models that classifies whether a pair of mentions are coreferent. It scores each candidate antecedent-mention relationship independently, often using a simpler feedforward network. The biaffine attention mechanism improves upon this by explicitly modeling the asymmetric relationship between a mention and its antecedent through a learned bilinear transformation.

Span Representation

A fixed-length vector encoding a contiguous sequence of tokens, typically computed by concatenating the hidden states of the span's start and end tokens from a model like SpanBERT. Biaffine attention operates over these span representations, computing a pairwise score that captures the compatibility between a mention's vector and a candidate antecedent's vector.

Higher-Order Inference

An iterative refinement technique where span representations are updated based on the representations of their predicted antecedents. Biaffine attention scores are recomputed over these updated representations, enabling the model to perform transitive reasoning across coreference chains. This allows the system to implicitly learn that if A=B and B=C, then A=C.

Antecedent Pruning

A computational efficiency technique that restricts the candidate antecedent search space before applying the biaffine scorer. Heuristic filters based on distance, syntactic constraints, and agreement features (e.g., number, gender) drastically reduce the number of pairs that need to be scored, making the quadratic complexity of biaffine attention tractable for long documents.

e2e-coref

The canonical end-to-end neural coreference resolution model by Lee et al. (2017) that popularized the use of learned span representations and biaffine attention. It jointly performs mention detection and coreference scoring, demonstrating that a single, unified architecture could outperform complex, multi-stage rule-based sieves on the CoNLL-2012 benchmark.

About the author

Prasad Kumkar

CEO & MD, Inference Systems

Prasad Kumkar is the CEO & MD of Inference Systems and writes about AI systems architecture, LLM infrastructure, model serving, evaluation, and production deployment. Over 5+ years, he has worked across computer vision models, L5 autonomous vehicle systems, and LLM research, with a focus on taking complex AI ideas into real-world engineering systems.

His work and writing cover AI systems, large language models, AI agents, multimodal systems, autonomous systems, inference optimization, RAG, evaluation, and production AI engineering.

Limited slotsGet a Free AI Consultation

How We Work

Custom AI workflows for your Business

One-fit-all AI don't work for modern businesses. At Inferensys, we aim to understand your business & custom requirements; which we use to define most efficient agentic workflows, the data, and the tools for your business.