Multicalibration

Unlike standard calibration, which only guarantees accuracy on the overall population, multicalibration ensures a predictor is calibrated on every subgroup C within a specified collection C. This collection can be generated by a computationally bounded adversary or an auditing algorithm, capturing complex, intersecting groups like 'Black women over 50' that simple demographic parity would miss. The guarantee holds simultaneously, preventing the model from hiding errors in minority slices.

The subgroups C are defined not just by protected attributes but by any computationally identifiable function. An auditor with limited compute power cannot find a subgroup where the predictor is miscalibrated. This connects fairness directly to computational complexity:

• A predictor is α-approximately multicalibrated if no efficient adversary can find a subgroup where calibration error exceeds α.
• This prevents 'fairness gerrymandering,' where a model appears fair on predefined groups but is biased on an ad-hoc intersection.

A multicalibrated predictor is a powerful omnipredictor: it can be used to optimally predict any downstream loss function from a given family. If a predictor is multicalibrated with respect to a rich class of functions, a simple post-processing of its output yields predictions that are no worse than the best function in that class for any convex loss. This transforms fairness from a constraint into a tool for robust transfer learning.

Multicalibration provides a rigorous framework for algorithmic auditing. An auditor iteratively searches for subgroups with high calibration error. If found, the predictor is updated to correct the error on that subgroup. This process:

• Converges quickly to a multicalibrated predictor.
• Produces an audit trail of discovered subgroups where the original model failed.
• Guarantees that no further computationally bounded search will find significant miscalibration.

Multicalibration inherently addresses intersectional fairness without requiring explicit enumeration of all intersecting groups. By allowing the adversary to define subgroups through arbitrary boolean functions over features, the guarantee automatically covers:

• High-dimensional intersections of protected attributes.
• Continuous features like income or geography combined with race.
• Learned subgroups discovered by a neural network auditor. This makes it strictly stronger than equalized odds or demographic parity on a fixed set of groups.

Multicalibration is a strengthening of multiaccuracy, which only requires the predictor's bias to be zero on average within each subgroup. Calibration further requires that among individuals receiving a predicted score of v, the true outcome proportion is also v. This distinction is critical:

• Multiaccuracy: 'The average prediction for women equals the average outcome.'
• Multicalibration: 'For women predicted to have a 20% probability, the true rate is exactly 20%.' This finer granularity prevents systematic over- or under-estimation within risk strata.

A fairness paradigm that examines bias against subgroups defined by the combination of multiple protected attributes (e.g., race and gender) rather than treating each in isolation. Multicalibration directly addresses intersectionality by requiring calibration on all computationally identifiable subgroups, including intersecting ones, preventing fairness gerrymandering where a model appears fair on single axes but discriminates against a specific intersection.

A class of fairness definitions that partition a population into groups defined by a protected attribute and require a statistical measure to be equal across them. Multicalibration is a stronger, more granular group fairness guarantee. While standard group fairness might only check broad categories like race or gender, multicalibration extends the guarantee to a rich, potentially infinite collection of subgroups defined by any computable function of the features.

A classic statistical property where a predictor's confidence scores match empirical probabilities. For example, among all instances receiving a 70% probability score, exactly 70% should be positive. Standard calibration only guarantees this on average over the entire population. Multicalibration strengthens this by requiring the property to hold simultaneously for every subgroup in a specified collection, preventing the model from being systematically overconfident or underconfident for any identifiable group.

A model debugging technique that computes performance metrics on semantically meaningful data subsets, or slices, to uncover performance disparities hidden by aggregate metrics. Multicalibration can be seen as an automated, exhaustive form of slicing analysis. Instead of manually defining slices to audit, a multicalibrated predictor guarantees that no slice in a computationally rich class exhibits miscalibration, providing a rigorous safety net for model auditors.

A systematic, often third-party evaluation of an algorithmic system to detect and measure discriminatory outcomes against protected groups. Multicalibration provides a provable audit framework: if an auditor can specify a class of subgroups they care about (e.g., all subgroups definable by a decision tree of depth 3), a multicalibrated model mathematically guarantees the absence of calibration bias within that entire class, moving auditing from spot-checking to comprehensive verification.

A causal definition of fairness where a prediction is fair if it remains the same in the actual world and a counterfactual world where the individual belonged to a different demographic group. While multicalibration is a statistical guarantee and counterfactual fairness is a causal one, they are complementary. A multicalibrated predictor can still violate counterfactual fairness if causal pathways are discriminatory, highlighting the need for layered fairness approaches in high-stakes deployments.