Fairness Indicators

Fairness Indicators provides an interactive UI that renders fairness metrics across multiple slices side-by-side. This allows for rapid, visual identification of performance disparities.

• Metric Selection: Toggle between false positive rate, false negative rate, accuracy, and other configurable metrics.
• Slice Comparison: Bar charts display metrics for each slice (e.g., 'race:Asian', 'gender:Female') against a baseline slice.
• Threshold Slider: Dynamically adjust the classification threshold to explore accuracy-fairness trade-offs in real-time.

The tool is designed as a native component of the TFX ecosystem, enabling fairness evaluation to be embedded directly into production machine learning pipelines.

• Evaluator Component: Fairness Indicators runs as part of the TFX Evaluator, comparing a candidate model against a baseline before deployment.
• Automated Validation: Pipeline owners can set threshold-based gating rules to block model promotion if fairness metrics fall below a defined bound.
• Model Card Generation: Evaluation results can be automatically exported to populate structured transparency documentation.

To compute fairness metrics, Fairness Indicators requires that each example in the evaluation dataset be annotated with its corresponding slice membership.

• Slice Keys: Data must include features that define protected groups, such as 'gender', 'age_group', or 'zip_code'.
• Counterfactual Ground Truth: The framework relies on the assumption that the ground truth label is independent of the sensitive attribute for a fair model.
• Multi-Valued Slices: Supports evaluation on intersectional subgroups by combining multiple slice keys into a single composite key.

The library computes a standard suite of group fairness metrics, each operationalizing a different mathematical definition of equity.

• Demographic Parity: Compares the positive prediction rate across slices.
• Equal Opportunity: Compares the true positive rate across slices.
• Equalized Odds: Compares both the true positive rate and the false positive rate across slices.
• Predictive Parity: Compares the positive predictive value (precision) across slices.

While deeply integrated with TensorFlow, the core metric computation logic is model-agnostic. It operates on a model's exported predictions and corresponding ground truth labels.

• Input Format: Requires a binary prediction score and a ground truth label for each example.
• Framework Flexibility: Can evaluate models built with PyTorch, JAX, or scikit-learn by exporting predictions to the expected format.
• Standalone Usage: The metric computation library can be used independently of the full TFX pipeline in a notebook environment for ad-hoc analysis.

The core mechanism of Fairness Indicators is slicing analysis, which decomposes aggregate performance metrics into semantically meaningful subgroups.

• Anomaly Detection: Surfaces hidden failures where a model achieves 99% overall accuracy but 50% accuracy on a small, critical slice.
• Intersectional Slicing: Evaluates combinations like 'race:Black AND gender:Female' to uncover biases masked by single-axis analysis.
• Data Distribution: The dashboard also visualizes the number of examples in each slice, alerting users to potential statistical insignificance.

A separation-based metric requiring a classifier to have equal True Positive Rates and equal False Positive Rates across different sensitive groups.

• Constraint: P(Ŷ=1|Y=1, A=a) = P(Ŷ=1|Y=1, A=b) AND P(Ŷ=1|Y=0, A=a) = P(Ŷ=1|Y=0, A=b)
• Fairness Indicators UI: Displays side-by-side confusion matrices for each slice, allowing direct comparison of TPR and FPR.
• Trade-off: Satisfying equalized odds and calibration simultaneously is impossible when base rates differ.

A causal definition where a prediction is fair if it remains the same in the actual world and a counterfactual world where the individual belonged to a different demographic group.

• Structural Causal Model: Requires a causal graph to distinguish discriminatory paths from legitimate explanatory paths.
• Audit Utility: Fairness Indicators can be extended to test counterfactual fairness by generating synthetic twins with flipped protected attributes.
• Contrast: Unlike observational metrics like demographic parity, this captures causal discrimination, not just correlation.

The observed tension where enforcing strict fairness constraints on a model can lead to a measurable reduction in overall predictive accuracy.

• Pareto Frontier: Fairness Indicators helps visualize this frontier by plotting accuracy against a fairness metric for different model thresholds.
• Mitigation: Techniques like Adversarial Debiasing and Reject Option Classification aim to find optimal points on this curve.
• Critical Question: The tool forces engineers to explicitly decide the acceptable cost of fairness in their specific context.