Attribute Inference Attack

The attack leverages spurious correlations between publicly available non-sensitive attributes (e.g., zip code, browsing history) and sensitive protected attributes (e.g., income, health status). The model inadvertently encodes these societal biases during training, and the attacker queries the model with known non-sensitive features to predict the unknown sensitive attribute. This differs from membership inference because the attacker does not need to know if the target was in the training set—only that the model reflects real-world statistical relationships.

The attacker operates under a black-box access scenario, requiring only API-level query access to confidence scores or predictions. The workflow proceeds as follows:

• Attacker possesses a target's non-sensitive feature vector (e.g., demographics, public social media data)
• Attacker queries the model with variations of this vector to observe output sensitivity
• A secondary shadow classifier is trained on model outputs to predict the sensitive attribute
• The attack succeeds when the model's predictions leak information about protected classes beyond what random guessing would yield

While model inversion aims to reconstruct representative training samples or class prototypes, attribute inference targets specific individuals. Key differences include:

• Attribute inference: Predicts a single sensitive feature of a known individual using their non-sensitive data as input
• Model inversion: Generates a synthetic face image or record that represents the average of a class (e.g., 'what does a typical person with high income look like?')
• Attribute inference is often more practically dangerous because it applies to specific, identifiable targets rather than statistical averages

Attribute inference has been demonstrated across multiple domains:

• Genomic privacy: Predicting undisclosed genetic markers from publicly shared ancestry data using models trained on genome-wide association studies
• Social networks: Inferring political affiliation, sexual orientation, or relationship status from friendship graphs and liked pages
• Healthcare: Deducing a patient's undisclosed condition from prescribed medications and procedure codes accessible through insurance APIs
• Financial services: Estimating income brackets from transaction metadata and merchant category codes processed by fraud detection models

Defending against attribute inference requires a layered privacy-preserving approach:

• Differential Privacy (DP): Training with DP-SGD ensures that the model's outputs do not depend significantly on any single individual's sensitive attributes
• Adversarial Representation Learning: Training encoders to explicitly remove sensitive attribute information from learned embeddings while preserving utility for the primary task
• Output Restriction: Limiting API responses to hard labels rather than confidence scores, or applying prediction clipping to bound information leakage
• Data Minimization: Excluding unnecessary sensitive attributes from training pipelines entirely when they are not required for the model's intended function

Quantifying attribute inference risk involves comparing attacker performance against baselines:

• Balanced accuracy of the attacker's classifier compared to a majority-class baseline
• Area Under the Curve (AUC) of the attacker's receiver operating characteristic curve
• Precision and recall at specific operating points relevant to the threat model
• Formal frameworks like privacy risk scores and inference advantage metrics measure how much the model improves an attacker's ability over prior knowledge alone
• Regular privacy audits using shadow model techniques should be conducted before model deployment