Robustness to Distillation

The watermark signal must be transferred from the teacher's soft labels to the student during distillation. This is achieved by embedding the watermark into the model's decision boundary rather than superficial output statistics. When a student model is trained to minimize KL divergence with the teacher's logits, the watermark's statistical signature propagates through the distillation process.

• Soft label transfer: Watermark patterns encoded in class probabilities survive temperature-scaled distillation
• Decision boundary invariance: Signatures embedded in the geometric structure of classification boundaries persist across architectural changes
• Logit distribution matching: The student learns to replicate the teacher's complete output distribution, including watermark-induced anomalies

The watermark is entangled with the model's internal feature representations rather than superficial output patterns. This entanglement forces any student model that successfully learns the teacher's task to also internalize the watermark signature.

• Representation binding: The watermark becomes inseparable from task-critical features learned during training
• Layer-wise activation patterns: Statistical signatures embedded across multiple network layers resist layer-wise distillation
• Mutual information maximization: The watermark and primary task features share high mutual information, making selective removal impossible without catastrophic forgetting

A carefully constructed trigger set of input-output pairs encodes the ownership signature. During distillation, the student model learns to reproduce these trigger behaviors because they appear as legitimate training examples within the teacher's output distribution.

• Indistinguishable triggers: Trigger samples are drawn from the same distribution as training data, preventing the adversary from filtering them out
• High-confidence mapping: Triggers produce overconfident predictions in the teacher, which the student prioritizes during knowledge transfer
• Statistical verifiability: The trigger set provides a null hypothesis test with a false positive rate below 10^-6, ensuring legal admissibility

The watermark payload is sized to match the information capacity of the student model. Even heavily compressed student architectures retain sufficient capacity to encode the ownership identifier when the payload is designed for the target compression ratio.

• Compression-adaptive encoding: Payload length scales with the expected capacity of the smallest anticipated student model
• Redundant embedding: The signature is embedded with error-correcting codes that tolerate partial information loss during aggressive distillation
• Bit error rate tolerance: Extraction succeeds with BER below 1% even when the student has 10x fewer parameters than the teacher

Distillation often uses temperature scaling to soften the teacher's output distribution. Distillation-robust watermarks maintain detectability across the full range of distillation temperatures by encoding signatures in the relative ordering of logits rather than their absolute magnitudes.

• Rank-order preservation: The watermark is encoded in the ordinal relationships between class probabilities, which survive temperature scaling
• Scale-invariant statistics: Detection metrics use normalized probability ratios that cancel out temperature effects
• Multi-temperature training: The teacher is trained with randomized temperature values during watermark embedding to ensure robustness across the entire distillation parameter space

The verification protocol does not assume any specific student architecture. Whether the adversary uses a smaller CNN, a transformer, or a completely different model family, the watermark remains extractable through black-box query access alone.

• Model-agnostic triggers: Trigger inputs are designed to produce consistent outputs regardless of the underlying architecture
• Cross-architecture transfer: Watermarks embedded via adversarial training generalize across heterogeneous model families
• Query-only extraction: Ownership can be verified through API access without requiring knowledge of the student's internal structure