Bias Detection

This foundational technique involves quantifying differences in model outputs across protected demographic attributes. It uses metrics to measure disparate impact and disparate treatment.

• Key Metrics: Calculate demographic parity (equal positive outcome rates across groups) and equalized odds (equal true positive and false positive rates).
• Example: Analyzing sentiment scores for names associated with different ethnicities or measuring occupation association strength (e.g., "nurse" vs. gender).
• Tooling: Libraries like fairlearn and AIF360 provide scikit-learn-compatible metrics for this analysis.

Systematic evaluation using curated datasets designed to probe for specific bias types. These benchmarks provide standardized, quantitative scores.

• Common Benchmarks: StereoSet (measures stereotype association), CrowS-Pairs (tests for social biases in masked language models), BOLD (Bias in Open-ended Language Generation Dataset), and Winogender (coreference resolution bias).
• Process: The model generates completions for benchmark prompts, and its outputs are scored against ground-truth non-biased references.
• Purpose: Allows for pre-deployment baselining and tracking improvement over model versions.

A causal technique that tests how model outputs change when protected attributes in the input are systematically altered, holding all else constant.

• Methodology: Create minimal pairs of inputs (e.g., "The doctor completed the surgery. He was..." vs. "The doctor completed the surgery. She was...") and compare the model's subsequent completions or probability distributions.
• Goal: Isolates the causal effect of the demographic marker on the output.
• Advantage: More granular than aggregate statistics, helping pinpoint specific linguistic contexts where bias manifests.

Examines bias encoded within the model's internal geometric representations (embeddings). This detects associations learned during pre-training.

• WEAT (Word Embedding Association Test): A statistical test measuring the differential association of target concepts (e.g., 'programmer', 'homemaker') with attribute concepts (e.g., 'male', 'female').
• SEAT (Sentence Embedding Association Test): Extends WEAT to sentence-level embeddings.
• Interpretation: Reveals implicit stereotypes (e.g., closeness of 'man' to 'career' and 'woman' to 'family' in vector space) that may propagate to downstream tasks.

Uses fill-in-the-blank or likelihood-scoring templates to measure the model's propensity for biased completions in controlled syntactic frames.

• Procedure: Use a large set of templates like "The [occupation] was very [adjective]." or "[Name] is from [country]. They are known for being [trait]."
• Measurement: Score the probability the model assigns to stereotypical vs. non-stereotypical adjective or trait fillers for different demographic slots.
• Strength: Highly scalable and automatable, providing a clear signal of association strength independent of prompt engineering.

Qualitative, adversarial testing where human auditors (often diverse panels) deliberately probe the model for biased outputs across sensitive domains.

• Red Teaming: Teams craft adversarial prompts targeting race, gender, religion, disability, and ideology to trigger and catalog harmful outputs.
• Crowdsourced Audits: Platforms like Dynabench facilitate large-scale, human evaluation of model outputs for subtle biases.
• Critical Role: Captures complex, intersectional, and context-dependent biases that purely automated metrics may miss, providing crucial ground truth for training safety classifiers.

Debiasing refers to the suite of techniques applied to reduce unwanted social biases in a model's outputs and internal representations. It is the corrective action taken after bias detection.

• Methods include: Counterfactual data augmentation, adversarial debiasing during training, and inference-time filtering.
• Key Distinction: Detection identifies the problem; debiasing attempts to solve it. Effective debiasing often requires retraining or fine-tuning.

Toxicity classification is the use of specialized machine learning models to automatically detect harmful, offensive, or abusive language. While related to bias, it focuses on explicit harm rather than implicit societal skew.

• Operational Focus: Identifies profanity, hate speech, harassment, and threats.
• Common Tools: Often implemented via APIs like the Perspective API or custom fine-tuned BERT models. It is a frequent component in a classifier chain for content moderation.

Explainable AI encompasses methods designed to make AI decisions interpretable to humans. For bias detection, XAI tools are critical for diagnosing why a model produced a biased output.

• Key Techniques: Feature attribution methods (e.g., SHAP, LIME) highlight which input words or features most influenced a biased prediction.
• Use Case: Auditors use XAI to trace a biased hiring recommendation back to specific phrasing in a job description or resume.

A safety benchmark is a standardized dataset and evaluation protocol used to measure and compare the safety and robustness of language models, including their propensity for bias.

• Examples: BOLD (Bias in Open-ended Language Generation), ToxiGen, and CrowS-Pairs are benchmarks specifically for bias evaluation.
• Purpose: Provides quantitative, reproducible metrics (e.g., sentiment disparity scores) to track model improvements or regressions across versions.

An algorithmic impact assessment is a systematic, pre-deployment evaluation of an AI system's potential risks, including bias, fairness, and societal effects. Bias detection is a core technical component within this broader governance process.

• Scope: Goes beyond technical metrics to consider legal compliance (e.g., EU AI Act), ethical guidelines, and stakeholder impact.
• Output: A risk rating and mitigation plan, often required for high-risk AI systems in regulated industries.

Red teaming is the proactive, adversarial testing of an LLM system to discover vulnerabilities, including biased outputs. It is a human-driven, exploratory form of bias detection.

• Process: Dedicated teams craft prompts designed to elicit stereotypes, discriminatory associations, or unfair refusals across protected attributes (gender, race, religion).
• Outcome: Generates qualitative failure cases that inform improvements to training data, model fine-tuning, and guardrails.