Representation Bias

The foundational mechanism of representation bias, occurring when the probability of selecting an instance from a subpopulation is not proportional to its true prevalence. This creates a non-i.i.d. training set that distorts the model's learned decision boundary.

• Under-coverage bias: Certain groups are systematically excluded from the sampling frame (e.g., a survey conducted only via smartphone misses populations without mobile access)
• Over-sampling artifacts: Minority groups may be artificially duplicated to achieve class balance, causing the model to memorize specific examples rather than learn generalizable patterns
• Truncation bias: Data collection stops at a threshold that disproportionately excludes one group (e.g., credit history minimums that filter out younger demographics)

Representation bias extends beyond raw data volume into the quality and consistency of annotations. When human labelers apply subjective judgments, their own demographic priors introduce systematic errors.

• Stereotypical association: Labelers may unconsciously assign negative sentiment to text written in African American Vernacular English (AAVE) at higher rates than to Standard American English expressing identical sentiment
• Definitional exclusion: The operational definition of a target variable may be calibrated on a majority group, making it an invalid measure for minority groups (e.g., creditworthiness proxies that rely on traditional banking history)
• Inter-rater reliability collapse: Annotation agreement scores may be high on majority-group examples but plummet on edge cases from underrepresented populations, masking data quality issues in aggregate metrics

When a subpopulation is underrepresented, the model's learned feature space fails to capture the variance within that group, leading to representation collapse where distinct individuals are mapped to indistinguishable embeddings.

• Within-group compression: The model allocates insufficient representational capacity to minority groups, treating them as a homogeneous cluster rather than a diverse distribution
• Proxy reliance amplification: With sparse data on a protected group, the model over-indexes on noisy proxy features (e.g., ZIP code as a stand-in for race) to make predictions, entrenching proxy discrimination
• Out-of-distribution fragility: Underrepresented groups are more likely to fall outside the model's learned manifold, triggering high-confidence but completely erroneous predictions

Representation bias compounds at the intersections of multiple demographic dimensions. A dataset may appear balanced on gender and race independently while catastrophically underrepresenting specific intersectional subgroups.

• Combinatorial sparsity: A dataset with 50% male/female and 50% Black/White representation may still have near-zero examples of Black women, creating a data desert at the intersection
• Masking by marginal distributions: Aggregate fairness metrics computed on single axes can report parity while intersectional subgroups experience severe discrimination
• Amplification in hierarchical models: When a model learns a compositional representation, bias at each level multiplies, causing intersectional subgroups to receive predictions that are worse than the sum of individual biases

Representation bias is not static; it evolves as the relationship between the training data distribution and the real-world population shifts over time. A dataset that was representative at collection time becomes progressively biased.

• Concept drift in demographics: Population demographics change, but the model's training distribution remains frozen, causing systematic underrepresentation of emerging or growing groups
• Feedback loop entrenchment: A biased model deployed in production influences which users engage with the system, skewing future training data toward already-overrepresented groups and creating a self-reinforcing bias cycle
• Seasonal and event-driven shifts: Data collected during specific temporal windows (e.g., holiday shopping patterns) may fail to capture behaviors of groups with different cultural or economic calendars

A subtle form of representation bias where the relationship between the measured features and the underlying construct differs across groups, rendering the model's internal logic invalid for underrepresented populations.

• Differential item functioning: A test question or feature may measure the intended construct in the majority group but measure a different construct entirely in a minority group (e.g., word familiarity tests that conflate vocabulary with cultural exposure)
• Construct bias: The theoretical construct itself (e.g., 'job performance') may be defined using metrics that are only valid for the majority group's work patterns, systematically misrepresenting minority group performance
• Metric invariance failure: Statistical tests for measurement invariance (configural, metric, scalar) fail across groups, indicating that the model's feature weights cannot be meaningfully compared or applied across populations