Fairness Metrics

A component of equalized odds that focuses exclusively on the True Positive Rate (TPR) , or recall. It computes the difference in TPR between groups, ensuring that qualified individuals have an equal chance of receiving a favorable outcome regardless of their sensitive attribute. A value of 0 means the model is equally good at finding true positives across groups.

• Formula: TPR(unprivileged) - TPR(privileged)
• Range: -1 to 1, with 0 being perfectly fair
• Use Case: Critical in hiring and lending where missing a qualified candidate is costly

Aggregates both Equal Opportunity Difference and False Positive Rate (FPR) difference into a single scalar metric. It calculates the average of the TPR difference and the FPR difference between groups. A value of 0 satisfies the equalized odds criterion, ensuring errors of both types are evenly distributed across all protected groups.

• Formula: 0.5 * [(FPR(unpriv) - FPR(priv)) + (TPR(unpriv) - TPR(priv))]
• Range: -1 to 1
• Advantage: Balances both sides of the error distribution, penalizing models that favor one group in one error type

A generalized entropy index borrowed from economics that measures the inequality in the distribution of model predictions or errors across groups. Unlike binary metrics, the Theil Index captures entropy-based divergence and is decomposable, meaning total unfairness can be broken down into within-group and between-group components. A value of 0 indicates perfect equality.

• Decomposability: Allows analysis of intersectional subgroups
• Range: 0 to ∞, with 0 being perfectly equal
• Advantage: Handles multi-class and continuous predictions natively

Evaluates whether a model's predicted probabilities are well-calibrated across groups. A model satisfies calibration by group if, for any predicted probability score, the fraction of positive outcomes is the same regardless of group membership. This metric measures the maximum deviation from perfect calibration, preventing systematic over- or under-estimation of risk for specific demographics.

• Requirement: P(Y=1 | ŷ=s, group=A) = P(Y=1 | ŷ=s, group=B) for all scores s
• Risk: Miscalibration leads to misallocated resources in credit scoring and medical triage
• Complement: Often used alongside equalized odds, as both cannot be satisfied simultaneously except in trivial cases

A group fairness metric requiring a model's positive prediction rate to be equal across all demographic groups. Also known as statistical parity, it ensures statistical independence from the sensitive attribute.

• Formula: P(Ŷ=1 | A=a) = P(Ŷ=1 | A=b)
• Measures whether different groups receive positive outcomes at the same rate
• Does not consider ground truth labels, only prediction distributions
• Commonly used in hiring and lending fairness audits

A fairness criterion requiring a classifier to have equal true positive rates and false positive rates across protected groups. This ensures errors are evenly distributed regardless of group membership.

• Matches both sensitivity and specificity across groups
• More stringent than demographic parity alone
• Requires access to ground truth labels for evaluation
• Key metric in recidivism prediction and medical diagnosis fairness

A legal and quantitative measure assessing whether a facially neutral policy disproportionately harms a protected group. Often evaluated using the 80% rule from EEOC guidelines.

• Formula: P(Ŷ=1 | unprivileged) / P(Ŷ=1 | privileged)
• A ratio below 0.8 indicates potential disparate impact
• Originates from US employment discrimination law
• Widely adopted in algorithmic fairness auditing frameworks

A comprehensive metric computing the average of the difference in false positive rates and true positive rates between privileged and unprivileged groups.

• Combines equal opportunity and predictive equality into one measure
• Value of 0 indicates perfect fairness by this definition
• Captures both types of classification errors simultaneously
• Used in AI Fairness 360 toolkit for model evaluation

A fairness criterion ensuring that a model's predicted probabilities accurately reflect the true likelihood of an outcome for every demographic group. Also called test-fairness or well-calibration.

• P(Y=1 | Ŷ=s, A=a) = s for all groups and scores
• Prevents systematic over- or under-estimation of risk
• Critical in credit scoring and insurance underwriting
• Can conflict with equalized odds in practice

An entropy-based measure from economics adapted to quantify inequality in model performance or prediction distribution across groups. Decomposable into between-group and within-group components.

• Captures both individual and group-level unfairness
• Decomposability enables granular bias diagnosis
• Higher values indicate greater inequality
• Useful for intersectional fairness analysis across multiple attributes