PATE

The foundational privacy mechanism of PATE relies on partitioning the sensitive training data into disjoint subsets. Each teacher model is trained exclusively on one partition, ensuring no single teacher has access to the entire dataset. This isolation means that even if an adversary compromises one teacher, they gain information about only a fraction of the records. The privacy guarantee strengthens as the number of teachers increases, because the student's query must aggregate knowledge distributed across many independent models.

The student model queries the teacher ensemble and aggregates their votes using the Confident-GNMax aggregator. This mechanism:

• Selects only teachers with high confidence in their predictions
• Adds carefully calibrated Gaussian noise to the vote counts
• Only reveals the winning label if the noisy margin exceeds a threshold This step is where the formal privacy budget (ε) is consumed. The noise scale and threshold are tuned to balance the privacy-utility trade-off, with tighter thresholds providing more utility at higher privacy cost.

Once trained, the teacher ensemble is used to label a large pool of unlabeled public data. The student model is then trained on this newly labeled dataset using standard supervised learning. This knowledge distillation step transfers the ensemble's collective intelligence to the student without the student ever accessing the sensitive data. The student learns only the aggregated, noisy consensus, which provides a strong membership inference defense because individual training records are never directly observed.

PATE's privacy guarantee is quantified using moments accountant tracking. Each query to the teacher ensemble consumes a portion of the privacy budget. The total privacy loss is bounded by tracking the Rényi Differential Privacy (RDP) parameters across all queries. A key advantage is that the student's training on public data incurs zero additional privacy cost—only the teacher queries count. This allows the student to be trained extensively without further eroding the privacy guarantee.

To minimize privacy leakage, PATE employs a selective answering strategy. The aggregator only returns a label when the teacher ensemble exhibits strong consensus. If the vote margin is too narrow or the teachers are uncertain, the query is rejected and no answer is given. This prevents the student from learning from ambiguous examples that would require many noisy queries to resolve, conserving the privacy budget for high-confidence, high-value labels that efficiently transfer knowledge.

PATE can be combined with virtual adversarial training (VAT) on the student to further smooth the decision boundary. By training the student to be robust to small input perturbations, the model learns to generalize from the noisy teacher labels without overfitting to the aggregation artifacts. This regularization technique improves the student's accuracy on the public data distribution while maintaining the formal differential privacy guarantee inherited from the teacher queries.

A collection of models trained on disjoint, non-overlapping partitions of the sensitive training data. Each teacher sees only a fraction of the total dataset, which is the foundational property that enforces privacy. No single teacher has access to all records, limiting the information that can leak about any individual. The ensemble typically uses identical architectures but diverges in parameters due to data isolation.

The mechanism that collects teacher votes on unlabeled public data and adds calibrated Gaussian noise before revealing the majority label. The GNMax aggregator:

• Counts teacher votes for each class
• Adds independent Gaussian noise to each vote count
• Returns the argmax only if the noisy plurality exceeds a threshold This ensures the student's label is differentially private with respect to the teacher training data.

A variant of the GNMax mechanism that only reveals the teacher ensemble's vote when there is strong consensus. It applies a threshold T to the noisy plurality score. If the highest noisy vote count minus the second-highest exceeds T, the label is revealed; otherwise, the query is discarded. This privacy amplification technique reduces the number of answered queries, tightening the overall privacy budget by limiting information release to high-confidence predictions.

The student model learns exclusively from the privacy-protected labels generated by the teacher ensemble on public, unlabeled data. The student never accesses the original sensitive training data. Training proceeds via:

• Semi-supervised learning if abundant public data exists
• Knowledge distillation using the noisy teacher votes as soft targets The student's architecture can differ from the teachers, enabling flexible deployment. Privacy guarantees transfer because the student only sees differentially private outputs.

PATE's privacy cost is bounded using moments accountant or Rényi Differential Privacy (RDP) composition. Each query to the teacher ensemble consumes a portion of the privacy budget. Key accounting factors:

• Number of answered queries (limited by Confident-GNMax threshold)
• Noise scale (σ₁ for vote counts, σ₂ for thresholding)
• Data-dependent analysis that tightens bounds when teachers agree The total (ε, δ)-differential privacy guarantee is computed post-training across all student queries.

An optional enhancement applied during student training that smooths the model's decision boundary around unlabeled data points. VAT adds a regularization term that penalizes the KL divergence between the student's predictions on clean inputs and adversarially perturbed inputs. This improves the student's generalization and robustness while indirectly reducing the risk of the student memorizing and leaking the noisy teacher labels, further strengthening the overall privacy posture.