Private Aggregation of Teacher Ensembles (PATE)

PATE operates on a knowledge distillation principle with two distinct model tiers:

• Teacher Models: Multiple models, each trained on a disjoint partition of the sensitive dataset. No single teacher sees all records.
• Student Model: Trained exclusively on the aggregated, noisy labels produced by the teacher ensemble, never accessing the original data.

This separation ensures that the student inherits the predictive power of the ensemble without direct exposure to sensitive training examples.

The core privacy guarantee comes from the aggregation step where teacher votes are counted and noise is injected:

• For each query from the student, teachers produce individual predictions.
• The Laplace mechanism adds calibrated random noise to the vote histogram.
• The noisiest label with the highest count is selected as the output.

This process satisfies differential privacy because the noisy output obscures the contribution of any single teacher and, by extension, any single training record.

PATE tracks privacy loss through a rigorous moment accountant that bounds the total information leakage:

• Each student query consumes a fraction of the privacy budget (epsilon).
• The accountant computes the Rényi differential privacy bound across all queries.
• Training halts when the predefined privacy limit is reached.

This provides a mathematically provable guarantee rather than heuristic privacy, making PATE suitable for regulated environments like healthcare and finance.

PATE leverages public, unlabeled data to bridge the gap between private teachers and the student:

• The student is trained on non-sensitive public data labeled by the teacher ensemble.
• This avoids wasting the privacy budget on queries that don't improve the student.
• The Gaussian noise scale is tuned based on the sensitivity of the teacher votes.

This design enables high-utility models even when the sensitive dataset is small, as the student benefits from abundant public data with privacy-preserving labels.

To optimize the privacy-utility tradeoff, PATE employs confident teacher voting:

• The student only receives labels when teachers exhibit strong consensus.
• Queries with divided teacher votes are discarded, conserving the privacy budget.
• A threshold parameter controls the required margin of agreement.

This filtering ensures that the student learns from high-confidence, stable predictions while avoiding wasteful expenditure of the privacy budget on ambiguous examples.

PATE provides robust protection against multiple attack vectors:

• Membership Inference: Noisy aggregation prevents adversaries from determining if a specific record trained a teacher.
• Model Inversion: The student never accesses raw features, only aggregated labels.
• Data Reconstruction: Disjoint teacher partitions prevent any single model from containing enough information to reconstruct full records.

These properties make PATE a defense-in-depth framework suitable for publishing models trained on highly sensitive data.

The mathematical foundation of PATE. Differential privacy provides a formal guarantee that the output of an analysis is essentially the same whether or not any single individual's data is included. PATE achieves this by having the student model learn only from noisy, aggregated teacher votes rather than raw data.

• Epsilon (ε): The privacy loss parameter; lower values mean stronger guarantees
• Mechanism: PATE adds calibrated Laplacian or Gaussian noise to teacher vote counts
• Composition: The total privacy budget is tracked across all student queries to bound cumulative leakage

The machine learning technique that PATE leverages for privacy-preserving transfer. In standard distillation, a compact student model learns to mimic a larger teacher ensemble. PATE repurposes this: the student never sees raw data, only the teachers' privacy-noised aggregated predictions.

• Soft labels: Teachers output probability distributions, not just hard class labels
• Temperature scaling: High-temperature softmax smooths teacher outputs, improving knowledge transfer
• Compression: The student can be a simpler, deployable model while teachers remain private

The primary privacy threat that PATE is designed to mitigate. In this attack, an adversary analyzes a model's outputs to determine whether a specific record was in its training set. PATE's disjoint teacher partitions and noisy aggregation make such inference statistically infeasible.

• Shadow models: Attackers train proxy models to calibrate membership signals
• Defense: Each teacher sees only a fraction of data, limiting per-model leakage
• Empirical evaluation: PATE's resistance is measured by attacker AUC-ROC against ground truth membership

An attack where an adversary reconstructs representative features of private training data by exploiting gradient information or confidence scores. PATE's architecture prevents direct access to teacher model parameters, closing a critical inversion vector.

• Gradient leakage: Standard federated learning can leak data through shared gradients; PATE shares only labels
• Confidence vector attacks: The student receives only the noised argmax or top-k votes, not full confidence distributions
• Defense-in-depth: Even if the student is compromised, teachers remain isolated and inaccessible

A quantifiable limit on total privacy loss that governs PATE's operation. Each student query consumes a fraction of the budget. Once exhausted, the student training must stop to maintain the formal guarantee.

• Moments Accountant: Tracks privacy loss across queries using advanced composition theorems
• Trade-off: Lower epsilon means more noise, potentially reducing student accuracy
• Selective answering: The student can choose not to query teachers when confidence is already high, conserving budget

A complementary cryptographic protocol that can harden PATE's aggregation step. SMPC allows teachers to jointly compute the noisy vote count without any single party seeing individual votes, adding a layer of cryptographic privacy on top of statistical privacy.

• Secret sharing: Teacher votes are split into shares and computed on distributively
• No trusted aggregator: Eliminates the need for a central server to see raw votes
• Combined guarantees: SMPC + differential privacy provides defense against both computational and statistical adversaries