Private Aggregation of Teacher Ensembles (PATE)

The foundational privacy guarantee stems from partitioning the sensitive dataset into non-overlapping shards. Each teacher model is trained exclusively on a single, isolated data partition. This ensures that no single teacher has access to the full dataset, limiting the information any individual model can leak. The privacy analysis relies on the sensitivity of the aggregation mechanism, which is bounded because each teacher's vote depends only on its own disjoint data shard.

When the student model queries for a label, each teacher casts a vote. The votes are tallied, and calibrated Laplacian noise is added to the vote counts. The label with the highest noisy count is selected. This mechanism satisfies (ε, δ)-differential privacy for each query. The privacy cost per query is determined by the noise scale and the number of teachers, with the composition theorem tracking the cumulative privacy budget spent as the student makes more queries.

The noisy labels generated by the teacher ensemble are used to train an unlabeled student model via knowledge distillation. The student never accesses the sensitive training data directly. A key property is post-processing immunity: once the noisy labels are produced with a differential privacy guarantee, any arbitrary processing of those labels—including training the student—cannot weaken the privacy guarantee. The student can be published or deployed without additional privacy risk.

PATE leverages privacy amplification by subsampling when the student selects a random subset of teachers for each query rather than polling the entire ensemble. This random subsampling introduces uncertainty about which teachers—and therefore which data partitions—contributed to a given answer. Combined with the noisy aggregation, this provides a significantly tighter privacy guarantee than querying all teachers, enabling more queries within the same privacy budget.

The PATE-GAN variant extends the framework to generative modeling. A differentially private generative adversarial network uses the PATE mechanism to train a discriminator with privacy guarantees. The discriminator acts as the teacher ensemble, trained on disjoint data partitions, and its noisy gradients guide the generator. This enables the creation of differentially private synthetic data that preserves the statistical properties of the sensitive source dataset without exposing individual records.

The Confident-GNMax mechanism improves the privacy-utility trade-off by only releasing a label when the teacher ensemble reaches a high-confidence consensus. It adds Gaussian noise to the vote counts and requires the plurality vote to exceed a threshold. If confidence is insufficient, the query is rejected, conserving the privacy budget. This selective answering strategy enables the student to learn from high-quality, privacy-protected labels while avoiding wasteful expenditure on ambiguous examples.

The core training paradigm where a compact student model is trained to mimic the behavior of a larger, more complex teacher model or ensemble. In PATE, the student learns from the aggregated, noisy votes of the teachers rather than directly from hard labels. The student's training objective is typically to minimize the cross-entropy between its output distribution and the aggregated teacher distribution, effectively transferring generalization capabilities without exposing the granular decision boundaries of the private teachers.

An extension of the PATE framework applied to generative adversarial networks. The discriminator is replaced by an ensemble of teacher discriminators trained on disjoint data partitions. These teachers vote on the realism of generated samples, and the votes are privately aggregated to train a student discriminator. This architecture allows the generation of high-fidelity synthetic data with formal differential privacy guarantees, as the generator never accesses real data directly.

The systematic tracking of cumulative privacy expenditure across multiple queries to the teacher ensemble. PATE uses moments accountant techniques to provide a tight bound on the total privacy loss (ε, δ). Each query to the aggregated teacher consumes a fraction of the privacy budget. The accountant tracks higher-order moments of the privacy loss random variable, enabling more accurate composition than basic sequential composition theorems and allowing practitioners to determine exactly how many student queries are permissible before the budget is exhausted.

The privacy-amplifying voting mechanism used in PATE to select the public label. The teachers vote, and the mechanism only reveals the label if the plurality vote exceeds a confidence threshold T. If the vote is not confident enough, the query is rejected. Gaussian noise is then added to the vote counts, and the max function is applied. This 'confident' step provides an additional layer of privacy by refusing to answer ambiguous queries that would require more information leakage to resolve, thus preserving the privacy budget for easier, high-consensus inputs.

The method by which the sensitive training dataset is divided into disjoint shards, each assigned to a single teacher model. The privacy guarantee relies on the fact that any single record influences at most one teacher. Common strategies include:

• Random partitioning: Each record is randomly assigned to a teacher.
• Stratified partitioning: Ensures each teacher's shard maintains the original class distribution.
• User-level partitioning: All records belonging to a single individual are assigned to the same teacher to provide user-level privacy guarantees.

A corpus of non-sensitive, publicly available data used to train the student model. The student queries the teacher ensemble with these unlabeled examples to obtain privacy-preserving pseudo-labels. The quality and distributional similarity of this public data to the private data is critical: if the public data is out-of-distribution, the student will query the teachers on inputs they cannot confidently classify, leading to high rejection rates or noisy labels that degrade student accuracy.