Word Embedding Debiasing

The foundational post-processing algorithm that identifies a gender subspace using a set of defining word pairs (e.g., he-she, man-woman). It performs two operations:

• Neutralize: Projects gender-neutral words like 'doctor' or 'nurse' onto the orthogonal complement of the gender subspace, removing their gender direction entirely.
• Equalize: Enforces that the only difference between inherently gendered word pairs (e.g., 'grandfather-grandmother') is their position along the gender direction, making them equidistant to all neutral words. This method explicitly removes direct bias but can leave indirect associations intact.

A linear classifier is trained to predict a protected attribute (e.g., gender) from the embeddings. The embeddings are then projected onto the nullspace of that classifier's weight matrix, removing all linear information about the attribute.

• The process repeats iteratively, training a new classifier on the projected embeddings and removing its nullspace, until the attribute can no longer be linearly predicted.
• This guarantees the removal of all linear bias but is computationally more intensive than single-step methods.

Uses Boolean matrix logic to softly remove a concept subspace from embeddings. A conceptor matrix is a positive semi-definite matrix that represents a linear subspace with a soft aperture.

• The operation C_not_gender = I - C_gender creates a conceptor that describes the space orthogonal to gender.
• Embeddings are multiplied by this conceptor, gently suppressing the gender component rather than hard-projecting it away.
• This approach is less destructive to semantic content than hard projection, preserving more of the original vector's utility.

A debiasing method that identifies a bias subspace via principal component analysis on the difference vectors of biased word pairs. It then applies a graded rotation:

• Words are rotated away from the bias subspace by an amount proportional to their original bias intensity.
• Unlike hard neutralization, this preserves variance in the neutralized dimensions, preventing the embedding space from collapsing.
• Particularly effective for debiasing contextualized embeddings from models like BERT without fine-tuning the entire model.

Extends static embedding debiasing to sentence-level representations from transformer models. It estimates a global bias subspace by aggregating the context-dependent representations of stereotypical sentences.

• The bias subspace is computed from the mean difference of sentence encodings for counterfactual pairs (e.g., 'He is a nurse' vs. 'She is a nurse').
• All subsequent sentence encodings are projected away from this subspace.
• This method addresses the challenge that bias in contextualized embeddings is dynamic and depends on the surrounding text.

A retraining approach rather than pure post-processing. It modifies the GloVe training objective to directly penalize the encoding of gender information in the embeddings of neutral words.

• An additional loss term minimizes the correlation between the gender direction and neutral word vectors during training.
• The result is a set of word embeddings that are born-fair, with gender bias never encoded in the first place.
• This avoids the information loss that can occur when surgically removing bias from a pre-trained space.