SISA Training

The training dataset is horizontally partitioned into mutually exclusive, non-overlapping shards. Each shard is assigned to an independent constituent model, ensuring that any single data point influences only one model replica. This isolation is the foundational mechanism that constrains the blast radius of a deletion request.

• Shards are typically created using random uniform partitioning
• Each shard trains a separate model instance from scratch
• No data point appears in more than one shard
• The number of shards directly trades off unlearning speed against model accuracy

Each shard trains a completely independent model replica with no weight sharing, gradient exchange, or distributed synchronization during training. This architectural isolation guarantees that when a deletion request arrives, only the single submodel that ingested the target data requires modification.

• Models are trained asynchronously on separate hardware
• No inter-model communication occurs during training
• Isolation eliminates cross-contamination between shards
• Enables parallel, non-blocking unlearning operations

Within each shard, training data is further divided into temporal slices and checkpoints are saved at slice boundaries. When unlearning is required, the affected submodel is rolled back to the checkpoint immediately preceding the slice containing the target data, then retrained only on the remaining slices.

• Slices act as intra-shard recovery points
• Reduces retraining cost from full shard to a subset of slices
• Checkpoint frequency determines the granularity of rollback
• Balances storage overhead against unlearning latency

At inference time, predictions from all constituent submodels are combined through an aggregation function to produce a single output. The ensemble nature of SISA provides built-in redundancy, as the removal of one submodel for retraining only marginally degrades overall performance during the unlearning window.

• Common aggregation: majority voting for classification
• For regression: arithmetic mean of submodel outputs
• Ensemble provides fault tolerance during unlearning
• Inference latency scales linearly with the number of shards

SISA provides exact unlearning—the retrained model is mathematically identical to one trained from scratch without the deleted data. Because each shard is independent, retraining a single shard on its remaining data produces a distribution that matches the gold standard of full retraining for that partition.

• No approximation error or statistical leakage
• Verifiable through membership inference attack auditing
• Satisfies strict regulatory requirements under GDPR Article 17
• Contrasts with approximate methods like gradient ascent

The primary cost of SISA is increased training overhead proportional to the number of shards, as each submodel must be trained independently. However, this upfront investment is amortized across the model lifecycle by dramatically reducing the cost of each subsequent unlearning operation.

• Training cost: S × cost of single model
• Unlearning cost: 1/S × cost of full retraining
• Storage overhead: S model checkpoints plus slice archives
• Optimal S balances training budget against expected deletion frequency

The foundational step of SISA training. The original dataset is horizontally partitioned into mutually exclusive, disjoint shards. Each shard is used to train an independent constituent model. This isolation ensures that a deletion request for a single data point affects only one shard, limiting the scope of retraining. The number of shards creates a direct trade-off between unlearning granularity and the overhead of managing multiple models.

Within each shard, data is further divided into temporal slices to create incremental checkpoints. The model for a shard is trained slice-by-slice, and its architecture and parameters are saved after each slice. When a deletion request arrives, the system can roll back to the checkpoint immediately before the target data was introduced, minimizing the computational cost of retraining to only the affected slices.

The final prediction is formed by aggregating the outputs of the independent constituent models. Common strategies include:

• Majority voting for classification tasks
• Averaging output probabilities or logits
• A trainable meta-model that learns to weight shard outputs This modular ensemble architecture ensures that the removal or retraining of one shard's model does not require touching the others.

SISA provides an exact unlearning guarantee. Because the data point to be deleted is contained within exactly one shard, retraining only that shard from the clean checkpoint produces a final aggregated model that is mathematically identical to one trained from scratch on the remaining dataset. This satisfies the strictest regulatory requirements for data removal without the prohibitive cost of full retraining.

The efficiency of SISA is governed by the sharding and slicing configuration:

• More shards: Faster unlearning (smaller retraining scope) but higher inference latency and storage costs.
• More slices: Finer granularity for rollback but increased checkpoint storage overhead. The optimal configuration balances deletion latency against operational costs for the specific deployment context.

While SISA partitions data into explicit, centrally managed shards, federated unlearning deals with removing a client's entire contribution from a global model trained across decentralized devices. SISA's approach is a centralized, deterministic strategy, whereas federated unlearning often relies on approximate methods or historical gradient storage. Both address the same core problem of data isolation for deletion but in fundamentally different architectural contexts.