Equalized Odds

Equalized Odds imposes two simultaneous constraints on a classifier's error profile:

• True Positive Rate Equality: The probability of a positive prediction given a truly positive instance must be equal across groups. This ensures equal recall or sensitivity.
• False Positive Rate Equality: The probability of a positive prediction given a truly negative instance must be equal across groups. This controls for unequal false alarm rates.

A model satisfying only the first condition meets the weaker Equal Opportunity criterion. Equalized Odds is stricter because it also requires the second, preventing a model from achieving parity in true positives by simply making more positive predictions for one group.

Demographic Parity requires the overall rate of positive predictions to be equal across groups, irrespective of ground truth. This can force equality of outcome, potentially harming accuracy by approving unqualified candidates from one group or rejecting qualified candidates from another.

Equalized Odds is a conditional criterion. It ties fairness to the actual qualification (ground truth). A model can have different overall approval rates under Equalized Odds, as long as those rates are justified by equal error rates. It is generally considered more aligned with merit-based decisions, as it seeks to equalize the quality of decisions, not their raw quantity.

For a binary classifier Ŷ, binary outcome Y, and protected attribute A (e.g., A ∈ {0,1}), Equalized Odds is satisfied if:

P(Ŷ = 1 | Y = y, A = 0) = P(Ŷ = 1 | Y = y, A = 1)

for all y ∈ {0,1}.

• When y = 1, this expresses True Positive Rate (TPR) equality: P(Ŷ=1 | Y=1, A=0) = P(Ŷ=1 | Y=1, A=1).
• When y = 0, this expresses False Positive Rate (FPR) equality: P(Ŷ=1 | Y=0, A=0) = P(Ŷ=1 | Y=0, A=1).

In practice, perfect equality is often impossible, so metrics like difference or ratio of these rates across groups are used to measure deviation from the ideal.

Under general conditions, key fairness criteria cannot be simultaneously satisfied. A fundamental result shows that, except in degenerate cases, a classifier cannot satisfy Equalized Odds and Calibration (where predicted scores reflect true probabilities) if the base rates of outcome Y differ across groups.

This creates a critical engineering trade-off:

• Enforcing strict Equalized Odds may require the model to be miscalibrated for some groups, meaning its confidence scores are no longer reliable probability estimates.
• Practitioners must decide which criterion—fairness in error rates or fairness in confidence estimation—is more critical for the specific application, as optimizing for one can degrade the other.

A common technique to impose Equalized Odds on an existing model is post-processing. This involves learning a separate transformation of the model's score or prediction for each demographic group.

The standard method, derived from Hardt et al. (2016), involves:

1. Using a validation set to compute the model's ROC curves for each group.
2. Finding group-specific decision thresholds that equalize TPR and FPR across groups, often by solving a linear program.
3. Applying these different thresholds at inference time based on group membership.

Advantage: No model retraining required. Limitation: Requires knowing the protected attribute at inference, which may be prohibited or raise legal concerns.

Ideal Use Cases:

• Criminal Justice Risk Assessment: Ensuring a tool's accuracy in predicting recidivism is equal across racial groups (equal TPR for 'high risk', equal FPR for 'low risk').
• Medical Diagnostic Tools: Guaranteeing a diagnostic AI has equal sensitivity (TPR) and specificity (1-FPR) for detecting a disease across gender or ethnicity.

Key Criticisms:

• Infringes on Individual Fairness: Satisfying group statistics can lead to unfair treatment of individuals within a group.
• Requires Group Labels: Enforcement necessitates collecting protected attributes, which can be invasive.
• Base Rate Ignorance: It does not account for differing prevalence of outcomes, which can make satisfying it statistically challenging or undesirable if the real-world rates truly differ.

In lending, equalized odds ensures that qualified applicants from all demographic groups have the same chance of approval (true positive rate), while unqualified applicants from all groups have the same chance of denial (false positive rate). This prevents models from being overly conservative or risky with specific populations.

• Example: A bank audits its model and finds a lower false positive rate for Group A (fewer unqualified applicants incorrectly approved) compared to Group B. To satisfy equalized odds, it must adjust thresholds to equalize these error rates, potentially increasing approvals for qualified members of Group B.
• Trade-off: Strict enforcement can reduce overall predictive accuracy, as the model is constrained by fairness, not just profit maximization.

Hiring tools use equalized odds to audit for gender or racial bias. The criterion requires that equally qualified candidates from all groups have the same probability of being shortlisted, and that unqualified candidates from all groups have the same probability of being incorrectly shortlisted.

• Implementation: A company discovers its model has a higher true positive rate for male candidates with certain degree keywords. An in-processing mitigation technique like adversarial debiasing can be applied during training to remove correlation between the model's latent representations and protected attributes like gender.
• Challenge: Defining "qualified" is critical; reliance on historical hiring data (which may be biased) to label training data can perpetuate historical bias.

Models predicting patient risk for readmission or disease onset must satisfy equalized odds to avoid disparities in care. It ensures that sick patients are equally likely to be flagged for intervention across groups, and that healthy patients are equally likely to be incorrectly flagged.

• Critical Application: A model used to allocate scarce healthcare resources (e.g., high-risk care management programs) must not systematically overlook at-risk patients from historically underserved populations.
• Post-processing Mitigation: For a deployed model, decision thresholds can be tuned separately for different demographic subgroups to achieve equalized odds without retraining, though this requires careful legal review.

Tools like COMPAS, which predict recidivism risk, are intensely scrutinized under fairness criteria like equalized odds. The goal is to ensure equal false positive rates (low-risk individuals incorrectly labeled high-risk) and true positive rates (high-risk individuals correctly identified) across racial groups.

• Famous Analysis: ProPublica's 2016 analysis alleged COMPAS violated equalized odds, showing different false positive rates between Black and white defendants. This highlighted the tension between different fairness definitions (e.g., demographic parity vs. equalized odds).
• Inherent Trade-off: It has been mathematically shown that, except in perfect prediction, equalized odds and demographic parity cannot be simultaneously satisfied if base rates (prevalence of the outcome) differ between groups.

When AI assists in selecting students, equalized odds audits ensure that truly deserving students from all backgrounds have an equal chance of selection, and that less-qualified students from all backgrounds have an equal chance of being mistakenly selected.

• Proxy Variable Risk: Even if race is excluded, features like high school name or standardized test scores can act as proxy variables for socioeconomic status and race, leading to indirect discrimination.
• Subgroup Analysis: Admissions officers perform intersectional analysis to evaluate equalized odds for subgroups like "low-income first-generation students" versus the general applicant pool, ensuring compounded disadvantages are addressed.

Regulators examine whether algorithmic pricing models satisfy fairness constraints. Equalized odds would require that high-risk customers pay higher premiums at equal rates across groups, and that low-risk customers receive lower premiums at equal rates.

• Conflict with Actuarial Fairness: Traditional insurance relies on risk-based pricing, which can lead to higher premiums for groups with historically worse outcomes. Equalized odds may conflict with this, posing a regulatory and ethical dilemma.
• Bias Audit Requirement: Under regulations like the EU AI Act, high-risk systems like insurance may require a formal bias audit and algorithmic impact assessment (AIA), with equalized odds as a key metric to be reported in a model card.

Equal opportunity is a foundational group fairness metric that requires a model's true positive rate (recall) to be equal across different demographic groups. It ensures that qualified individuals from each group have an equal chance of receiving a positive outcome (e.g., loan approval, job interview).

• Key Difference from Equalized Odds: Equalized odds is a stricter condition. While equal opportunity only constrains the true positive rate, equalized odds additionally requires equality of the false positive rate. A model can satisfy equal opportunity but fail equalized odds if its error rates differ between groups.

Demographic parity (also called statistical parity) is a group fairness criterion that requires the overall selection rate or proportion of positive predictions to be identical across groups, independent of individual qualifications.

• Comparison to Equalized Odds: These are often conflicting goals. Demographic parity ignores model accuracy and the actual prevalence of a qualified condition in each group. Enforcing it can force a model to make less accurate predictions. Equalized odds, by contrast, is conditioned on an individual's actual label (e.g., qualified/not qualified), making it more compatible with accuracy-based optimization.

Disparate impact is a legal doctrine and form of algorithmic bias that occurs when a facially neutral model produces outcomes that disproportionately and adversely affect a protected group. It is typically measured using the 80% rule (or four-fifths rule), where the selection rate for a protected group must be at least 80% of the rate for the most favored group.

• Relation to Fairness Metrics: Disparate impact is closely related to the technical metric of demographic parity. A finding of disparate impact often triggers a requirement to demonstrate business necessity. Achieving equalized odds can be a technical strategy to justify a model's use by showing its predictions are conditionally accurate across groups.

A fairness constraint is a formal mathematical condition integrated into a model's training or post-processing to enforce a specific definition of equity. Equalized odds is one such constraint.

• Implementation: In-processing techniques incorporate equalized odds as a penalty or a hard constraint within the loss function, forcing the optimizer to balance accuracy with equalized error rates. Post-processing methods adjust decision thresholds per group on a trained model's score outputs to satisfy the equalized odds condition. Common optimization frameworks include reductions approach or adversarial training.

Subgroup analysis is the practice of evaluating model performance metrics separately for distinct demographic slices. Intersectional analysis extends this by examining subgroups defined by combinations of multiple protected attributes (e.g., race and gender).

• Critical for Equalized Odds: Auditing for equalized odds is fundamentally a subgroup analysis task. It requires calculating true positive and false positive rates for each protected group. Intersectional analysis is crucial, as a model may satisfy equalized odds for broad categories like "gender" but fail for intersectional groups like "Black women," where bias is compounded. Tools like slice-based evaluation or disaggregated evaluation are used.

In-processing bias mitigation refers to techniques applied during model training to reduce unfair discrimination. Equalized odds is a target for several advanced in-processing methods.

• Adversarial Debiasing: A primary predictor is trained for the main task (e.g., loan default) while an adversarial network tries to predict the protected attribute from the primary model's representations. This removes information correlated with the protected attribute.
• Constrained Optimization: Algorithms like Reductions (Agarwal et al.) treat equalized odds as a set of constraints and solve a constrained empirical risk minimization problem, often finding a randomized classifier that satisfies the constraints.
• These methods directly shape the model's learned parameters to satisfy the equalized odds criterion.