LLM Bias Evaluation

Evaluates how a model portrays different demographic groups, focusing on the perpetuation of negative stereotypes, denigration, or erasure. Unlike allocative harm (denying resources), this measures the symbolic injury caused by biased outputs.

• Stereotype Detection: Uses curated benchmarks like StereoSet to measure the association between social groups and attributes.
• Toxicity Probing: Employs tools like Perspective API on model completions to quantify the generation of hate speech or profanity.
• Erasure Metrics: Checks if the model systematically fails to recognize or generate text about specific identities.

Quantifies how model accuracy and capability vary across demographic cohorts. A model exhibits disparate performance if its error rate for one group significantly exceeds that of another, even if the task seems neutral.

• Cohort Accuracy: Measures F1 scores or perplexity for distinct dialect speakers (e.g., African American Vernacular English vs. Standard American English).
• Named Entity Recognition (NER) Gap: Tests if the model fails to identify names associated with specific ethnicities.
• Co-reference Resolution: Evaluates if the model correctly links pronouns to non-binary or gender-neutral antecedents.

A causal approach that swaps sensitive attributes in a prompt while holding all other semantics constant. If the model's output changes significantly based solely on the demographic signifier, bias is present.

• Template Substitution: Replaces {name} or {pronoun} tokens in a fixed template (e.g., '{Name} is a doctor. Describe {pronoun}.') to isolate the effect of identity.
• Perturbation Sensitivity: Measures the divergence in sentiment or toxicity scores between the original and counterfactual completions.
• Benchmarks: Utilizes datasets like WinoBias or BOLD (Bias in Open-Ended Language Generation Dataset).

A qualitative, human-driven or automated process of attacking the model with prompts specifically designed to elicit edge-case failures and harmful outputs that automated metrics miss.

• Jailbreak Attempts: Crafting prompts to bypass safety guardrails and force the model to generate toxic or biased content.
• Multi-Turn Attacks: Testing if a benign conversation can gradually steer the model toward a biased persona over multiple exchanges.
• Domain-Specific Harms: Probing for biased medical advice or legal analysis that could cause material harm to specific groups.

Analyzes the vector representations learned by the model to detect latent biases encoded in the semantic relationships between concepts.

• Word Embedding Association Test (WEAT): Measures the cosine similarity between sets of target words (e.g., career terms) and attribute words (e.g., gender terms) to quantify stereotypical associations.
• Sentence Embedding Bias: Projects sentence-level embeddings to visualize if neutral contexts (e.g., 'a person working as a...') cluster differently based on the profession's stereotyped gender.
• De-biasing Projections: Identifies specific linear directions in the embedding space that encode bias for potential removal.

Assesses whether the model's subjective certainty aligns with its actual accuracy across groups. A miscalibrated model may be overconfident in wrong answers for specific demographics.

• Expected Calibration Error (ECE): Bins predictions by confidence and measures the gap between confidence and accuracy for each demographic slice.
• Selective Prediction: Evaluates if the model knows when to abstain from answering, ensuring it doesn't confidently generate hallucinated or biased facts for under-represented groups.
• Log-Probability Analysis: Examines the raw token probabilities to see if the model assigns lower likelihood to factual statements about minority figures.

The foundational discipline of designing machine learning systems that make impartial decisions, avoiding unjust bias against individuals or groups based on protected attributes. In the context of LLMs, this extends beyond classification parity to include representational harms—how models stereotype, denigrate, or erase entire demographic groups through generated text. Key principles include:

• Group fairness: Statistical parity across demographic segments
• Individual fairness: Similar treatment for similar individuals
• Causal fairness: Decisions independent of sensitive attributes in counterfactual worlds

Quantitative measures used to evaluate and monitor the presence and magnitude of bias in model outputs. For LLM evaluation, these metrics are adapted to assess disparate performance across demographic groups on tasks like question-answering, toxicity detection, and sentiment analysis. Common metrics include:

• Statistical parity difference: Gap in positive prediction rates between groups
• Equalized odds difference: Disparity in true positive and false positive rates
• Counterfactual token fairness: Probability shifts when swapping demographic terms in prompts
• Toxicity score variance: Differences in generated toxicity when prompted with identity terms

A causal definition of fairness where a decision is considered fair if it would remain the same in a counterfactual world where an individual's sensitive attributes were different. In LLM evaluation, this translates to counterfactual prompt augmentation:

• Systematically swapping identity terms (e.g., names, pronouns, demographic markers) in prompts
• Measuring the divergence in model outputs across counterfactual pairs
• Detecting subtle representational harms where sentiment, competence assumptions, or toxicity shift based solely on demographic signifiers This approach is central to benchmarks like BOLD and HONEST, which use templated counterfactuals to quantify stereotypical associations.

An in-processing bias mitigation technique that trains a model to simultaneously predict a target variable while an adversarial network attempts to predict the protected attribute from the model's internal representations. The primary model is optimized to maximize task accuracy while minimizing the adversary's ability to recover sensitive information. For LLMs, this technique is applied during fine-tuning:

• An adversarial classifier is attached to hidden layers to predict demographic attributes
• The language model is penalized for representations that encode sensitive information
• The resulting model learns fair representations that are invariant to protected attributes This approach is particularly effective for reducing representational harms in text generation.