AI Fairness 360 (AIF360)

Provides over 70 fairness metrics to test for bias across binary and multi-class classification, as well as regression tasks.

• Group fairness metrics: Statistical parity difference, disparate impact, equal opportunity difference, average odds difference, and Theil index.
• Individual fairness metrics: Consistency score and generalized entropy index.
• Supports both privileged and unprivileged group designation for structured bias testing.

Includes a library of pre-processing, in-processing, and post-processing algorithms to reduce unwanted bias.

• Pre-processing: Reweighing, Optimized Preprocessing, and Disparate Impact Remover transform training data to remove discrimination before modeling.
• In-processing: Adversarial Debiasing and Prejudice Remover add fairness constraints directly into the model training objective.
• Post-processing: Equalized Odds Postprocessing and Calibrated Equalized Odds adjust predicted labels after training.

Uses a consistent StructuredDataset class that wraps Pandas DataFrames with explicit metadata about protected attributes, privileged classes, and favorable labels.

• Eliminates ambiguity in how demographic features are encoded.
• Enables seamless interoperability between all metrics and mitigation algorithms.
• Includes built-in loaders for benchmark fairness datasets like Adult Census Income, COMPAS Recidivism, and German Credit.

Provides an Explainer module and visualization utilities to generate interactive dashboards for comparing model performance across groups.

• Visualize metric trade-offs between accuracy and fairness.
• Disaggregate performance by protected attribute values.
• Export reports suitable for bias audit documentation and stakeholder review.

Designed with a modular architecture that allows researchers and engineers to contribute custom fairness metrics and bias mitigation algorithms.

• All components inherit from base classes, ensuring new additions automatically integrate with existing pipelines.
• Supports integration with popular ML frameworks including scikit-learn, TensorFlow, and PyTorch.
• Active open-source community with regular contributions from academic and industry researchers.

An open-source Microsoft toolkit that complements AIF360 by providing fairness assessment dashboards and mitigation algorithms. While AIF360 focuses on a vast library of metrics, Fairlearn emphasizes interactive visualization and the accuracy-fairness trade-off curve.

• Provides ExponentiatedGradient and GridSearch for constrained optimization.
• Integrates natively with scikit-learn pipelines.
• Offers a drag-and-drop dashboard for disaggregated performance analysis.

A legal doctrine and the statistical foundation for the Four-Fifths Rule. In AIF360, this is operationalized via the BinaryLabelDatasetMetric class. A model exhibits disparate impact if the ratio of favorable outcomes for a protected group relative to the advantaged group falls below 0.8.

• AIF360 metric: disparate_impact.
• Originates from U.S. employment law (Title VII of the Civil Rights Act).
• Remains a primary regulatory trigger for bias audits.

A stricter fairness criterion than statistical parity. It requires that a classifier achieves the same true positive rate and false positive rate across all protected groups. AIF360 implements this via equalized_odds_difference.

• Penalizes models that are accurate overall but make different types of errors for different groups.
• Often used in recidivism prediction and credit scoring contexts.
• Mathematically constrains both sensitivity and specificity simultaneously.

An in-processing mitigation technique available in AIF360. It trains a predictor and an adversary simultaneously. The predictor tries to maximize accuracy on the target task, while the adversary tries to predict the protected attribute from the predictor's output.

• The predictor is penalized if the adversary succeeds.
• Results in a representation that is maximally non-discriminatory.
• Effective for removing indirect bias encoded in latent features.

A pre-processing algorithm in AIF360 that assigns weights to training examples to ensure fairness before model training. It calculates weights based on the expected probability of observing a specific combination of a protected attribute and a label.

• Does not require modifying the classifier itself.
• Computationally lightweight compared to in-processing methods.
• Effective for removing direct statistical dependencies but may not address intersectional bias.

A post-processing technique that modifies the decision boundary around a region of high uncertainty. When a prediction falls within a confidence band around the classification threshold, the algorithm flips the decision for the unprivileged group to a favorable outcome.

• Only applicable to binary classification problems.
• Preserves the original model architecture.
• Useful when retraining the model is impossible due to cost or black-box vendor constraints.