Bias Detection

Disparate impact analysis is a statistical technique that measures whether a model's outcomes disproportionately affect different demographic groups, regardless of intent. It calculates the ratio of favorable outcome rates between a protected group (e.g., a specific race) and a privileged group.

• Key Metric: The four-fifths rule (or 80% rule) is a common legal benchmark, where a ratio below 0.8 may indicate adverse impact.
• Process: Analysts stratify model predictions (e.g., loan approvals) by protected attribute and compare approval rates.
• Limitation: It measures correlation, not causation, and can be sensitive to small sample sizes in minority groups.

This technique involves calculating standard performance metrics separately across subgroups defined by protected attributes to surface unequal performance.

• Equal Opportunity: Checks if the true positive rate is similar across groups. A model that misses true positives for one group more than another violates this.
• Predictive Parity: Examines if the precision (positive predictive value) is equal across groups.
• Example: A facial recognition system with 99% accuracy for lighter-skinned males but 65% accuracy for darker-skinned females exhibits a clear fairness metric disparity. Tools like Fairlearn and AI Fairness 360 automate these calculations.

Counterfactual fairness is a causal reasoning approach that asks: "Would the model's prediction change if the individual's protected attribute (e.g., gender) were different, while all other relevant attributes remained the same?"

• Method: Generate synthetic or perturbed data points where only the protected attribute is altered.
• Goal: A fair model should produce the same prediction for both the actual and counterfactual individual.
• Use Case: Testing a resume screening model by creating counterfactual resumes identical in skills and experience but with different gender-indicating names, then observing score changes.

Adversarial debiasing is an in-training detection and mitigation technique where a secondary adversarial network attempts to predict the protected attribute from the primary model's embeddings or predictions.

• Mechanism: The primary model is trained for its main task (e.g., credit scoring) while simultaneously being trained to fool the adversary, making its internal representations uninformative for guessing the protected attribute.
• Outcome: This minimizes the encoding of bias-related information in the model's latent space, reducing its ability to make discriminatory decisions.
• Detection Role: The adversary's success rate during training serves as a direct, real-time measure of how much bias information the model retains.

This technique analyzes the internal vector representations (embeddings) learned by a model to uncover stereotypical associations or unintended subgroup clustering.

• Process: Extract embeddings for input data (e.g., job titles, names) and use dimensionality reduction (like t-SNE or UMAP) to visualize them.
• Detection: Bias is indicated if embeddings cluster strongly by protected attributes unrelated to the task. For example, in a resume embedding space, if "nurse" and "receptionist" vectors cluster near female-associated names while "engineer" and "CEO" cluster near male-associated names, it reveals learned societal bias.
• Metric: Average Cosine Similarity between group centroids can quantify separation.

Stress testing systematically perturbs input data along protected dimensions to observe unstable or skewed model behavior.

• Techniques:
  • Name Swapping: Replacing typically gendered or ethnic names in text inputs.
  • Attribute Masking: Redacting protected attributes to see if predictions change.
  • Template-Based Tests: Using sentence templates (e.g., "[Person] is a [profession]") and filling them with different group identifiers.
• Goal: To identify contextual bias where model outputs change inappropriately based on demographic cues. This is a key component of adversarial testing pipelines for LLMs and classifiers.

Also known as statistical parity or group fairness. This metric assesses whether a model's positive prediction rate is equal across different demographic groups. It is defined as P(Ŷ=1 | A=a) = P(Ŷ=1 | A=b) for all groups a, b.

• Key Insight: It focuses solely on the outcome, not on the accuracy of those outcomes.
• Limitation: Can be satisfied by an uninformed model that makes random predictions, and it does not account for potential differences in group prevalence or qualification rates.
• Use Case: Screening processes where the goal is equal selection rates, independent of underlying base rates.

A fairness criterion requiring that the true positive rate (recall) is equal across groups. Formally, P(Ŷ=1 | Y=1, A=a) = P(Ŷ=1 | Y=1, A=b).

• Focus: Ensures qualified individuals from each group have an equal chance of being correctly identified.
• Contrast with Equalized Odds: Equal Opportunity is a relaxation of the stricter Equalized Odds, which requires both true positive rates and false positive rates to be equal across groups.
• Example: In a loan approval model, this ensures that among all actually creditworthy applicants, the approval rate is the same for different demographic groups.

Also known as outcome test. This metric evaluates whether the precision (positive predictive value) of a model is equal across groups. It asks: given a positive prediction, is the likelihood of it being correct the same for everyone? Formally, P(Y=1 | Ŷ=1, A=a) = P(Y=1 | Ŷ=1, A=b).

• Key Insight: Focuses on the accuracy of positive predictions.
• Limitation: It is mathematically impossible to satisfy Predictive Parity and Equal Opportunity simultaneously if the base rates (prevalence of Y=1) differ between groups (except in perfect predictors). This is known as the fairness impossibility theorem.
• Use Case: Situations where the cost of a false positive is high and must be consistent, such as in certain medical diagnostics.

A legal and statistical test originating from U.S. employment law (the 80% rule or four-fifths rule) used to identify adverse impact. It calculates the ratio of the selection rate for a protected group to the selection rate for the most favored group.

• Calculation: (Selection Rate for Group A) / (Selection Rate for Group B). A result less than 0.8 often indicates potential disparate impact.
• Legal Context: Unlike intent-based disparate treatment, disparate impact concerns unintentional discrimination caused by a facially neutral policy or algorithm.
• Statistical Test: Often accompanied by a chi-squared test or Fisher's exact test to determine if the observed disparity is statistically significant.

Formal statistical tests used to determine if observed differences in model performance or outcomes between groups are statistically significant or likely due to random chance.

• Chi-Squared Test: Used for categorical outcomes (e.g., approval/denial) to test for independence between group membership and the model's decision.
• T-Test / ANOVA: Used for continuous outcomes (e.g., risk scores) to test for significant differences in the mean scores across groups.
• Kolmogorov-Smirnov Test: A non-parametric test used to compare the entire distribution of scores (e.g., probability outputs) between two groups, detecting differences in shape, spread, and central tendency.
• McNemar's Test: Used on paired nominal data, often to compare error rates (e.g., false positives) between two groups on the same set of examples.

A causal fairness notion that asks: Would the model's prediction for an individual have been the same if their protected attribute (e.g., race or gender) were different, while keeping all other relevant, non-discriminatory factors constant?

• Causal Framework: Requires building a causal model of the data-generating process to identify which variables are mediators (influenced by the protected attribute) and which are resolvers (independent).
• Key Requirement: Predictions must be based on variables that are not descendants of the protected attribute in the causal graph, unless those descendants are themselves considered fair to use.
• Strength: Moves beyond correlations to reason about what-if scenarios, aiming for fairness at the individual level.
• Challenge: Requires strong assumptions and domain knowledge to specify a valid causal model.

The process of identifying when a generative AI model, particularly a large language model, produces confident but factually incorrect or nonsensical information not grounded in its source data. This is distinct from bias but equally critical for factual integrity.

• Key Methods: Include retrieval-augmented generation (RAG) consistency checks, embedding similarity against source documents, and citation verification.
• Example: A model inventing a non-existent historical event or citing a fake academic paper would be flagged by hallucination detection systems.

The automated identification of language that is rude, disrespectful, hateful, or otherwise likely to harm a user. It often uses specialized machine learning classifiers trained on labeled datasets of toxic speech.

• Common Categories: Harassment, hate speech, threats, and severe profanity.
• Implementation: Typically deployed as a content filter that scores text and blocks or flags outputs exceeding a safety threshold. This works in tandem with bias detection to ensure outputs are not only fair but also civil.

The identification of attempts to manipulate a language model by embedding malicious instructions within its input, aiming to override its original system prompt and intended behavior. This is a security-focused validation.

• Attack Vectors: User inputs containing commands like "Ignore previous instructions" or hidden code.
• Defensive Techniques: Include input sanitization, anomaly detection on prompt structure, and separating user data from system instructions using frameworks like the Model Context Protocol (MCP).

Personally Identifiable Information (PII) detection is the automated scanning of data streams or AI outputs to identify sensitive information like names, social security numbers, email addresses, or credit card numbers.

• Purpose: Critical for privacy compliance with regulations like GDPR and HIPAA.
• Methods: Uses pattern matching (regex), named entity recognition (NER) models, and context analysis. It ensures outputs do not inadvertently leak private data, complementing bias detection's role in ethical output.

Deterministic checks that an output conforms to a predefined technical and business structure.

• Schema Validation: Verifies that structured data (e.g., JSON, XML) matches required fields, data types, and formats.
• Business Rule Validation: Applies explicit logical rules (e.g., "total cost must equal sum of line items").
• Role: Provides a foundational layer of correctness for agentic tool-calling and API execution, ensuring outputs are usable by downstream systems before more complex semantic checks like bias detection are applied.

The identification of rare items, events, or observations that deviate significantly from the majority of the data or from an expected pattern. In validation, it acts as a broad, unsupervised safety net.

• Use Case: Flagging outputs that are statistical outliers in terms of length, sentiment, embedded vector location, or metadata, which may indicate a novel type of error, bias, or adversarial attack not covered by other specific detectors.
• Techniques: Include statistical models, autoencoders, and clustering algorithms that learn a "normal" distribution of outputs during a validation pipeline's training phase.