Distillation Attack

The attacker queries the watermarked teacher model with a large corpus of unlabeled data to collect soft labels—probability distributions over classes rather than hard classifications. These probability vectors contain rich dark knowledge about the teacher's decision boundaries. The student model is then trained exclusively on these extracted input-output pairs, never accessing the original proprietary weights. This black-box approach requires no internal access to the victim model, making it a practical threat for models exposed via public APIs.

The core mechanism that enables watermark removal is signal attenuation during knowledge transfer. The student model learns to mimic the teacher's general decision boundaries but does not precisely replicate the specific weight configurations or statistical biases that encode the watermark. Key factors influencing attenuation:

• Temperature scaling: Higher softmax temperatures smooth probability distributions, diluting the watermark's trigger set responses
• Capacity mismatch: A smaller student model lacks the representational capacity to memorize the teacher's watermark artifacts
• Ensemble distillation: Averaging soft labels from multiple teachers further obscures any single model's fingerprint

A successful distillation attack must balance watermark removal against functional fidelity. The attacker's objective is to produce a student model that:

• Maintains comparable accuracy on the original task benchmark
• Eliminates detectable traces of the embedded ownership identifier
• Survives statistical correlation tests designed to detect the watermark

This creates a fundamental tension: aggressive distillation that thoroughly removes the watermark often degrades task performance, while conservative distillation that preserves accuracy may leave detectable watermark remnants. Attackers optimize this trade-off through careful temperature tuning and architecture selection.

Modern watermarking schemes incorporate specific defenses against distillation attacks:

• Entangled watermarking: Embedding the signature deep within feature representations essential to task performance, so removal destroys utility
• Trigger set hardening: Designing backdoor triggers that are robust to the temperature smoothing applied during distillation
• Multi-bit payloads: Encoding complex ownership messages that require high-fidelity reconstruction, making partial removal detectable
• Statistical redundancy: Distributing the watermark across many parameter layers so that distillation cannot uniformly erase the signal

Distillation attacks range from naive to highly sophisticated:

• Naive distillation: Direct training on teacher soft labels without counter-forensic awareness
• Adversarial distillation: Incorporating explicit anti-watermark objectives into the student's loss function to actively suppress known fingerprint patterns
• Multi-stage extraction: Using intermediate surrogate models to progressively dilute the watermark across successive distillation generations
• Hybrid fine-tuning: Combining distillation with targeted pruning and weight perturbation to attack both white-box and black-box watermarks simultaneously

Each variant represents an escalating arms race between watermark designers and model thieves.

Distillation attacks create significant challenges for intellectual property enforcement. When a stolen model is distilled, the original watermark may be sufficiently degraded to fall below the statistical significance threshold required for legal proof of ownership. This undermines:

• Copyright infringement claims: Weakened watermark correlation reduces evidentiary value
• DMCA takedown requests: Requires demonstrable proof of unauthorized copying
• Trade secret litigation: Distillation blurs the line between independent creation and misappropriation

Forensic investigators must employ cumulative evidence strategies, combining residual watermark detection with behavioral fingerprinting and training data provenance analysis.