Bias in Data

Historical bias occurs when past societal inequities and prejudices are captured and perpetuated in the training data. This bias reflects real-world disparities, meaning the data is an accurate but unjust representation of reality.

• Example: A hiring model trained on decades of industry data may learn to undervalue resumes from historically underrepresented groups, as the data reflects past discriminatory hiring practices.
• Challenge: This bias is embedded in the ground truth labels of the data, making it particularly insidious and difficult to correct without fundamentally re-evaluating the target variable.

Representation bias, or sampling bias, occurs when the training data does not adequately reflect the diversity of the population or scenarios the model will encounter in production.

• Causes: Non-random data collection, under-sampling of edge cases, or geographic limitations.
• Example: A facial recognition system trained primarily on images of lighter-skinned individuals will have higher error rates for darker-skinned faces.
• Impact: Leads directly to poor model performance and higher error rates on underrepresented subgroups, which can be identified through subgroup analysis.

Measurement bias arises from systematic errors in how data is collected, labeled, or quantified. The instruments or processes used introduce a consistent skew.

• Data Collection: Using sensors with different calibrations or accuracies.
• Labeling Subjectivity: Human annotators applying inconsistent or culturally biased labels. For instance, labeling emotions in speech can vary significantly across cultures.
• Proxy Variables: Using an imperfect or correlated measure for the true construct of interest (e.g., using credit score as a sole proxy for financial trustworthiness).

Aggregation bias occurs when a single model is applied uniformly across diverse groups, ignoring meaningful subgroup differences. It assumes data from all groups can be lumped together without losing critical heterogeneity.

• Example: A healthcare diagnostic model trained on a general population may fail for a specific ethnic group if that group exhibits different biological baselines or disease presentations.
• Link to Fairness: This bias is a primary cause of disparate impact, where a one-size-fits-all model creates disproportionately adverse outcomes for certain groups. Intersectional analysis is crucial for detecting it.

Evaluation bias seeps in when the benchmark datasets or metrics used to assess a model's performance are themselves flawed or non-representative. A model can appear unbiased because it was tested on biased ground truth.

• Benchmark Issues: Using test sets that lack diversity or contain the same historical biases as the training data.
• Metric Selection: Relying solely on aggregate accuracy, which can mask poor performance on minority subgroups. This highlights the necessity of fairness metrics like equal opportunity.
• Consequence: Creates a false sense of model robustness and fairness before deployment.

These data biases are the primary inputs that lead to algorithmic unfairness in model outputs. The pipeline is direct:

1. Biased Data (Historical, Representation, etc.) is used for training.
2. The model learns and amplifies these patterns.
3. Outputs manifest as disparate impact or disparate treatment.

Mitigation must therefore target the appropriate stage:

• Pre-processing: Correcting representation or measurement bias in the dataset.
• In-processing: Using techniques like adversarial debiasing to prevent the model from learning biased correlations.
• Post-processing: Adjusting decision thresholds for different groups to achieve demographic parity or equalized odds.

Historical bias originates from pre-existing societal inequities and prejudices that are captured in the training data. This bias reflects real-world patterns of discrimination or exclusion, meaning a model trained on such data will learn and perpetuate these skewed patterns.

• Example: A hiring model trained on decades of industry data may learn to deprioritize candidates from demographics historically underrepresented in that field, not due to qualification but due to past hiring practices.
• Key Challenge: This bias is embedded in the recorded facts of the dataset, making it a fundamental issue of garbage in, garbage out. Mitigation often requires conscious curation or synthetic augmentation of data.

Representation bias occurs when the training data does not adequately reflect the full diversity of the population or scenarios the model will encounter in production. This leads to poor model performance on underrepresented groups or edge cases.

• Mechanism: If a facial recognition system is trained primarily on images of individuals with lighter skin tones, its accuracy will be lower for individuals with darker skin tones.
• Core Issue: It is a failure of data collection and sampling. Aggregate performance metrics can mask severe performance drops for specific subgroups, which is why subgroup analysis is critical.

A proxy variable is a feature within a dataset that is statistically correlated with a protected attribute (e.g., race, gender). Even if the protected attribute is explicitly removed from the training data, a model can learn to discriminate by using these proxies.

• Common Examples: ZIP/postal code can proxy for socioeconomic status and race; job title or university name can proxy for gender.
• Detection & Mitigation: Identifying proxies requires feature analysis and techniques like adversarial debiasing, where a model is trained to make its internal representations uninformative for predicting the protected attribute.

Disparate impact is a legal and fairness concept describing a situation where a model's outputs, while facially neutral in design, have a disproportionately adverse effect on members of a legally protected group. It focuses on outcomes, not intent.

• Quantification: Often measured using the four-fifths (80%) rule, where the selection rate for any protected group should be at least 80% of the rate for the group with the highest selection rate.
• Contrast with Disparate Treatment: Unlike disparate treatment, where a protected attribute is used directly, disparate impact arises from seemingly neutral factors that act as proxies. It is a primary target for bias audits and regulatory compliance.

Bias mitigation encompasses technical interventions applied at different stages of the ML pipeline to reduce unfair discrimination.

• Pre-processing: Techniques applied to the training data itself. This includes reweighting samples from different groups or transforming features to remove correlation with protected attributes.
• In-processing: Techniques applied during model training. This involves adding fairness constraints (e.g., for demographic parity) to the loss function or using adversarial debiasing.
• Post-processing: Techniques applied to model predictions after training. The most common method is adjusting decision thresholds separately for different demographic groups to meet a fairness metric, without retraining the model.

Subgroup analysis is the practice of evaluating a model's performance metrics separately for distinct population slices (e.g., by gender, age bracket). This reveals performance disparities hidden by aggregate metrics like overall accuracy.

• Intersectional analysis extends this by evaluating performance across subgroups defined by the combination of multiple protected attributes (e.g., Black women over 50). This is crucial because bias can be compounded at these intersections.
• Tooling: Frameworks like Fairlearn and AIF360 provide standardized methods for this analysis. It is a foundational step in any comprehensive bias audit or Algorithmic Impact Assessment (AIA).