Disparate Impact

The cornerstone of disparate impact is a facially neutral policy or model. This means the system's rules, features, or algorithms do not explicitly reference a protected attribute like race, gender, or age. The discrimination is not overt but emerges from the application of the policy.

• Example: A hiring algorithm that scores candidates based on "cultural fit" or "pedigree" (e.g., university ranking) may inadvertently disadvantage applicants from underrepresented backgrounds, even though those terms are not directly protected categories.

A disparate impact claim requires demonstrating a statistically significant disproportionate adverse effect on a protected group. This is typically measured by comparing the selection rate or positive outcome rate between groups.

• The 80% Rule (Four-Fifths Rule): A common regulatory heuristic from U.S. employment law. If the selection rate for a protected group is less than 80% of the rate for the most favored group, it may indicate adverse impact.
• Statistical Tests: More rigorous audits use tests like chi-squared or Fisher's exact test to determine if the observed disparity is unlikely due to random chance.

Disparate impact often occurs through proxy variables—seemingly innocuous features that are highly correlated with a protected attribute. The model learns to discriminate via these proxies, even when the protected attribute is excluded from the training data.

• Common Proxies:
  • Zip/Postal Code (correlates with race/ethnicity and socioeconomic status).
  • Shopping Patterns or Brand Affinities.
  • Vocabulary Use or Typing Speed in assessments.
  • Network Graph Features (e.g., connections on a professional platform).

Under legal frameworks like Title VII in the U.S., a finding of disparate impact is not automatically unlawful. The deploying organization can mount a business necessity defense. This requires proving the challenged practice is job-related and consistent with business necessity.

• Burden of Proof: The defendant must show the model's specific features are essential for predicting a legitimate business outcome (e.g., job performance, credit risk).
• Less Discriminatory Alternative: The plaintiff can still prevail by demonstrating there is an alternative practice that serves the same business necessity with less discriminatory impact.

It is crucial to distinguish disparate impact from disparate treatment. Disparate treatment is intentional discrimination where a protected attribute is used explicitly to make decisions.

• Disparate Treatment: A loan model uses "gender" as an input feature to assign different interest rates.
• Disparate Impact: A loan model uses "occupation" and "zip code," leading to systematically higher rejection rates for women because those features act as proxies. The intent may be absent, but the harmful outcome is present.

Identifying disparate impact requires moving beyond aggregate metrics. It is detected through rigorous subgroup analysis, slicing evaluation data by protected attributes to compare performance metrics like false positive rates, precision, and recall.

• Intersectional Analysis: The most severe impacts often occur at the intersection of multiple attributes (e.g., Black women, older workers with disabilities). Analysis must examine these compounded subgroups.
• Fairness Metrics: Detection relies on metrics like demographic parity, equal opportunity, and equalized odds to quantify the disparity.

Disparate treatment is a form of algorithmic bias where a model explicitly uses a protected attribute (e.g., race, gender) as a direct input feature to make different decisions for different groups. This is often considered intentional discrimination.

• Key Difference from Disparate Impact: Disparate treatment is about explicit use, while disparate impact is about disproportionate outcomes from facially neutral rules.
• Example: A hiring algorithm that uses 'gender' as a feature and systematically ranks female candidates lower.
• Legal Context: Both are prohibited under U.S. employment law (Title VII), but they require different forms of evidence and legal defense.

A proxy variable is a feature in a dataset that is highly correlated with a protected attribute, allowing a model to effectively discriminate even when the protected attribute is excluded. This is a primary mechanism through which disparate impact occurs.

• Common Examples: Zip code (for race or socioeconomic status), university name (for legacy admissions), shopping patterns.
• Technical Challenge: Detecting and mitigating proxy bias is difficult because the correlation is often complex and non-linear.
• Mitigation: Techniques include fairness constraints and adversarial debiasing to learn representations that are invariant to the protected attribute.

Fairness metrics are quantitative measures used to assess whether a model's predictions are equitable across demographic subgroups. They provide the mathematical basis for detecting disparate impact.

• Demographic Parity: Requires the overall rate of positive predictions (e.g., loan approvals) to be equal across groups. Directly measures disparate impact.
• Equal Opportunity: Requires the true positive rate (recall) to be equal across groups. Focuses on not harming qualified individuals.
• Equalized Odds: A stricter criterion requiring both true positive rates and false positive rates to be equal across groups.
• Usage: These metrics often conflict with each other and with accuracy, requiring trade-off analysis.

A bias audit is a systematic, documented evaluation of an AI system to detect, measure, and report on potential discriminatory biases. Auditing for disparate impact is a central component.

• Process: Involves subgroup analysis to calculate fairness metrics across protected groups and intersectional analysis for compounded effects.
• Tools: Often conducted using fairness toolkits like IBM's AI Fairness 360 (AIF360) or Microsoft's Fairlearn.
• Outputs: Findings are documented in artifacts like Model Cards, which transparently report performance and fairness limitations.
• Regulatory Context: Mandated by laws like New York City's Local Law 144 for automated employment decision tools.

Post-processing bias mitigation involves techniques applied to a trained model's outputs to reduce disparate impact, without modifying the model itself. This is often the most practical approach for deployed systems.

• Primary Method: Adjusting decision thresholds (the cutoff score for a positive prediction) independently for different demographic groups to achieve a target fairness metric like demographic parity or equalized odds.
• Advantage: Does not require retraining the model, making it fast to implement.
• Disadvantage: Can reduce overall accuracy and may not address root causes of bias in the model's internal representations.
• Use Case: Correcting a credit scoring model's approval rates after a bias audit reveals a disparate impact.

An Algorithmic Impact Assessment (AIA) is a structured, policy-guided process to evaluate the potential risks and societal effects of deploying an automated system. Assessing risk of disparate impact is a core objective.

• Scope: Broader than a technical bias audit; includes stakeholder consultation, review of system purpose, and analysis of procedural justice.
• Frameworks: Guided by frameworks like the EU AI Act's risk-based approach or the Algorithmic Accountability Act.
• Output: A report detailing potential harms (including disparate impact), mitigation strategies, and plans for ongoing monitoring.
• Goal: Proactive governance to prevent harm, rather than just technical measurement post-deployment.