Private Aggregation of Teacher Ensembles (PATE)

The foundational privacy mechanism of PATE relies on splitting the sensitive training data into disjoint subsets. Each teacher model is trained exclusively on one partition, ensuring no single model has access to the full dataset. This limits the maximum possible privacy leakage from any individual teacher, as each model's knowledge is confined to a strict subset of records. The partitioning strategy directly impacts the privacy-utility trade-off; more partitions increase privacy but reduce the training data available per teacher.

The student model queries the teacher ensemble for predictions on unlabeled public data. The aggregation mechanism, typically the Confident-GNMax aggregator, selects the majority vote only if it exceeds a strict threshold and adds calibrated Gaussian noise. This step is the formal source of differential privacy. Key properties:

• Thresholding: Only high-consensus votes are revealed, suppressing low-confidence outputs that leak more information.
• Gaussian Noise: Noise is added to the vote counts, calibrated to the sensitivity of the aggregation function.
• Privacy Amplification: The consensus check and noise injection amplify the privacy guarantee beyond what individual teacher training provides.

The student model is trained on the noisy, aggregated labels generated by the teacher ensemble on a public, unlabeled dataset. The student never accesses the original sensitive data or the individual teacher models. This process transfers the ensemble's collective knowledge into a single, deployable model. The student's training loss is computed against the noisy labels, and the resulting model can be publicly released with a quantifiable privacy guarantee derived from the aggregation mechanism's parameters.

PATE's privacy guarantee is analyzed using moments accountant techniques, which tightly bound the total privacy loss over multiple queries to the teacher ensemble. Each query to the teachers consumes a portion of the privacy budget. The analysis tracks the privacy loss random variable's moments to compute the final (ε, δ)-differential privacy parameters. This accounting is critical for determining when to stop answering student queries, as the privacy budget is finite and must be formally enforced.

A practical extension of PATE involves training the student with a semi-supervised learning approach. The student is trained on a combination of the noisy teacher-labeled public data and a large corpus of unlabeled public data using techniques like Virtual Adversarial Training (VAT) or MixMatch. This significantly improves the student's accuracy without requiring additional queries to the teacher ensemble, thereby conserving the privacy budget while boosting model utility.

The teacher ensemble does not answer every student query. The Confident-GNMax aggregator only returns a label when the noisy consensus exceeds a predefined threshold. Queries that fail the threshold are discarded, and the student receives no answer. This selective answering mechanism conserves the privacy budget by only spending it on high-confidence, high-utility predictions. Once the cumulative privacy loss reaches the pre-defined budget limit, all further queries are halted, and the student model is finalized.

The core training paradigm underlying PATE, where a compact student model is trained to mimic the behavior of a larger, more complex teacher ensemble. The student learns not from raw data labels, but from the output probability distributions (soft labels) of the teachers.

• Transfer Mechanism: The student generalizes by matching the teacher's class probabilities, capturing dark knowledge about inter-class similarities.
• PATE's Innovation: Replaces standard teacher outputs with noisy, differentially private votes to prevent the student from memorizing sensitive teacher training data.

A mathematical guarantee that the output of a computation is statistically indistinguishable whether or not any single individual's data is included. PATE achieves this by injecting noise into the teacher voting process.

• Privacy Budget (ε): A quantifiable metric controlling the total privacy loss. A smaller epsilon indicates stronger privacy.
• Mechanism: PATE typically uses the Laplace Mechanism or Gaussian Mechanism to add calibrated noise to teacher vote counts before the student sees them.

The architectural prerequisite for PATE's privacy guarantee. The sensitive training dataset is split into disjoint, non-overlapping partitions.

• Teacher Isolation: Each teacher model is trained exclusively on a single partition, ensuring no single teacher has access to the full dataset.
• Privacy Amplification: The aggregation step combines knowledge from isolated models. The privacy guarantee relies on the fact that any individual record influences at most one teacher's vote.

The privacy-preserving voting mechanism at the heart of PATE. Instead of a simple majority vote, the system collects teacher predictions and adds carefully calibrated random noise to the vote counts.

• Query Restriction: The student can only access the aggregated, noisy labels, never the individual teacher votes or the underlying data.
• Privacy Accounting: Each query to the teacher ensemble consumes a portion of the privacy budget. The Moments Accountant is used to tightly track cumulative privacy loss across all student queries.

An advanced PATE variant that replaces the student model with a Generative Adversarial Network (GAN) to generate synthetic labeled data instead of training a classifier directly.

• Mechanism: The PATE teacher ensemble labels a small set of unlabeled public data. A GAN is then trained to generate new, synthetic labeled examples conditioned on these noisy labels.
• Benefit: Allows unlimited training of downstream models on the synthetic data without additional privacy cost, as the GAN's output is protected by the post-processing immunity of differential privacy.

The rigorous tracking of cumulative privacy loss during student training. Every time the student queries the teacher ensemble, the privacy guarantee degrades.

• Rényi Differential Privacy (RDP): Often used for tighter composition analysis than standard (ε, δ)-DP, providing a more accurate accounting of the total privacy spend.
• Fixed Budget Enforcement: Training must halt once the pre-defined epsilon threshold is reached to maintain the formal privacy guarantee. This requires careful balancing of model accuracy against the number of permitted queries.