Bias Audit

A defensible audit begins with a formal charter that explicitly defines the protected attributes, fairness criteria, and subpopulations under examination. This scope must be established before any data analysis begins to prevent p-hacking or metric shopping. The charter should specify:

• The legal and ethical basis for selecting protected groups (e.g., race, gender, age)
• The quantitative fairness definitions to be measured (e.g., demographic parity, equalized odds)
• The decision threshold for flagging a disparity as actionable
• The independence of the audit team from the model development team

Aggregate metrics across broad demographic categories can mask severe harm at the intersections. A rigorous audit employs slicing analysis to compute fairness metrics on combinatorially defined subgroups (e.g., Black women over 60) rather than treating each protected attribute in isolation. This approach, grounded in the theory of intersectional fairness, reveals latent bias that would otherwise be invisible. Modern tooling like Fairness Indicators enables the systematic computation of performance across hundreds of intersecting slices.

A naive audit that only checks for the direct use of protected attributes misses the critical problem of proxy discrimination. The audit must systematically test whether non-protected features (e.g., zip code, browser type, educational institution) serve as effective stand-ins for protected attributes. Techniques include:

• Computing mutual information between each feature and the protected attribute
• Training a proxy prediction model to assess how easily the protected attribute can be reconstructed from ostensibly neutral features
• Auditing the model under fairness through unawareness to demonstrate its insufficiency

Statistical parity alone cannot distinguish between discriminatory and legitimate causal pathways. A rigorous audit incorporates causal fairness frameworks using structural causal models and directed acyclic graphs to map the relationships between protected attributes, mediating variables, and outcomes. This enables the auditor to answer counterfactual questions: Would this individual have received the same prediction if they belonged to a different demographic group, holding all legitimately relevant qualifications constant? This is the foundation of counterfactual fairness.

A point-in-time audit is insufficient for a deployed system. The evaluation must assess the model's fairness characteristics over time to detect feedback loops—the phenomenon where a model's biased predictions influence the future training data, creating a self-reinforcing cycle of amplification. The audit should simulate the system's dynamics under different deployment conditions and establish a monitoring plan with drift detection thresholds for fairness metrics, not just accuracy metrics.