Gender Shades

The audit revealed a stark accuracy gap at the intersection of race and gender. While classifiers performed well on lighter-skinned males, error rates increased significantly for darker-skinned females. The maximum error rate for lighter-skinned males was less than 1%, while the error rate for darker-skinned females exceeded 34% in the worst-performing system. This demonstrated that evaluating bias on a single axis—such as gender alone—masks the compounded discrimination experienced by intersectional subgroups.

All three evaluated commercial classifiers—IBM Watson, Microsoft Cognitive Services, and Face++—exhibited the same directional bias. Performance degraded consistently when classifying darker-skinned females compared to lighter-skinned males. The specific error rate gaps between these two subgroups were:

• IBM: 22.3 percentage point gap
• Microsoft: 20.8 percentage point gap
• Face++: 33.8 percentage point gap This uniformity suggested a systemic failure in training data diversity rather than an isolated algorithmic flaw.

By disaggregating results, the audit isolated skin type as a stronger predictor of error than gender alone. The performance drop from lighter to darker skin types was more pronounced than the drop from male to female subjects. Key findings:

• Classifiers were 11.8% to 19.2% more accurate on lighter individuals overall.
• Gender classification error rates for darker-skinned females were consistently the highest across all vendors. This finding challenged the prevailing industry assumption that gender bias was the primary fairness concern in facial analysis.

The Gender Shades findings directly influenced the creation of model cards and datasheets for datasets, which are now standard transparency documentation practices. The audit demonstrated that:

• Disaggregated evaluation across intersectional subgroups is essential for meaningful bias detection.
• Reporting aggregate accuracy alone conceals harmful performance disparities.
• Independent, third-party auditing is necessary because self-reported vendor benchmarks often exclude challenging demographic subgroups. The paper has been cited in regulatory frameworks, including the EU AI Act's emphasis on testing high-risk systems for discriminatory impacts.

A critical methodological contribution was the systematic annotation of subjects using the Fitzpatrick skin type classification system, a dermatological scale originally designed to assess UV sensitivity. The audit team developed a protocol for annotating facial images with Fitzpatrick types I through VI, enabling reproducible skin-type analysis. This approach has since been adopted by subsequent fairness audits and influenced the development of more phenotypically diverse training datasets, though it has also sparked debate about the limitations of using a unidimensional scale to capture the complexity of human skin tone.

A framework for evaluating algorithmic bias that examines how overlapping social identities—such as race and gender—combine to create unique, compounded experiences of discrimination. The Gender Shades project is the seminal case study for this concept, demonstrating that evaluating gender or skin type in isolation masks severe failure modes at their intersection. Standard single-axis fairness metrics are insufficient to detect these multiplicative harms.

A form of data bias occurring when the training dataset underrepresents or fails to adequately cover certain segments of the population. Gender Shades traced the root cause of intersectional failure directly to this issue: the benchmark facial recognition datasets used for training (like IJB-A and Adience) were overwhelmingly composed of lighter-skinned male subjects. This led to phenotypic bias in model performance.

A quantitative measure used to evaluate the presence and magnitude of bias in a model's predictions. Gender Shades used disaggregated accuracy across intersectional subgroups as its primary metric. Key metrics include:

• Demographic Parity Difference: Compares positive prediction rates across groups
• Equalized Odds: Compares true positive and false positive rates
• Accuracy Parity: Compares overall correctness across subgroups

A systematic, independent evaluation of an algorithmic system to detect and measure discriminatory outcomes against protected groups using quantitative fairness metrics. The Gender Shades study is the archetypal external bias audit, conducted by researchers without access to proprietary models or training data. It established the methodology of using intersectional benchmark datasets to stress-test commercial APIs.