Split CIFAR

The benchmark is built upon the CIFAR-100 dataset, which contains 60,000 32x32 color images across 100 fine-grained classes (e.g., 'apple', 'lawn_mower', 'maple_tree').

• Task Splitting: The 100 classes are divided into a sequence of tasks. The most common configurations are 10 tasks of 10 classes each or 20 tasks of 5 classes each.
• Disjoint Classes: Each task contains a unique, non-overlapping set of classes. A model never sees images from a future task's classes during the training of a current task.
• Standardized Protocol: This fixed construction allows for direct, apples-to-apples comparison between different continual learning algorithms published in research.

The primary purpose of Split CIFAR is to quantify a model's susceptibility to catastrophic forgetting. After training on the final task, the model is evaluated on a held-out test set containing examples from all 100 classes seen throughout the sequence.

• Average Accuracy: The key metric is the final average accuracy across all classes. A high score indicates successful knowledge retention.
• Forgetting Measure: Researchers often track the drop in accuracy for each task from its peak performance after initial learning to its final performance. The average of these drops quantifies overall forgetting.
• Baseline Comparison: Performance is compared against two bounds: Joint Training (upper bound, where the model sees all data mixed together) and Fine-Tuning (lower bound, where the model is simply fine-tuned on each new task, leading to severe forgetting).

Split CIFAR is most frequently used to evaluate the challenging Class-Incremental Learning (CIL) scenario.

• No Task ID at Test Time: During evaluation, the model is not told which task a test image belongs to. It must classify among all 100 possible classes it has ever seen.
• Expanding Output Layer: The model's classification head must grow to accommodate new classes. A common approach is to add new output neurons for each new task.
• Major Challenge: This requires the algorithm to solve both stability (protecting old knowledge) and plasticity (learning new classes) while managing an ever-growing set of potential outputs. Algorithms like iCaRL were specifically designed for this setting.

Split CIRAR serves as a proving ground for the three main families of continual learning algorithms:

• Regularization-Based Methods: Algorithms like Elastic Weight Consolidation (EWC) and Synaptic Intelligence (SI) add penalty terms to the loss function to protect important parameters for old tasks. Their effectiveness is tested on the visual feature interference in Split CIFAR.
• Replay-Based Methods: Techniques like Experience Replay (ER) and Generative Replay store or generate past examples. Split CIFAR tests their memory efficiency—can good performance be maintained with a small buffer (e.g., 2000 images) for 10-20 tasks?
• Architectural Methods: Strategies like Progressive Neural Networks dynamically expand the model. Split CIFAR measures their parameter efficiency and ability to prevent forward/backward interference between the visual tasks.

Split CIFAR is part of a family of split-dataset benchmarks. Understanding its relatives clarifies its specific role:

• Split MNIST/CIFAR-10: A simpler variant splitting 10 classes into 5 tasks of 2 classes. Often used for initial prototyping.
• Split TinyImageNet: A more difficult benchmark using the 200-class TinyImageNet dataset, presenting a greater challenge in representation learning and forgetting.
• Split CIFAR-100 with Domain Shifts: Some variants introduce domain shifts within the sequence (e.g., changing image styles) to create a Domain-Incremental Learning challenge on top of the class-incremental one.
• Streaming/Online Split CIFAR: A stricter evaluation where the model sees each training example only once in a streaming fashion, testing online continual learning algorithms.

While a standard, Split CIFAR has known limitations that inform research directions:

• Artificial Task Boundaries: Data arrives in clear, discrete chunks, which is often not the case in real-world streaming data.
• Balanced and I.I.D. within Tasks: Data within a task is independent, identically distributed, and balanced across classes, unlike natural data streams which are often imbalanced and correlated.
• Focus on Classification: It measures a narrow slice of intelligence (multi-class image classification). It does not evaluate continual learning for reinforcement learning, language modeling, or multi-modal tasks.
• Fixed, Small-Scale Data: The dataset is relatively small. Forgetting dynamics can differ on larger-scale datasets like continual versions of ImageNet.

Average Accuracy is the primary metric, calculated as the mean classification accuracy across all tasks after the final training phase. It provides a holistic view of overall performance but can mask forgetting if early task performance degrades significantly. For a benchmark like Split CIFAR-100 with 10 tasks, it is computed as:

• ACC = (1/T) * Σ (Accuracy on Task i after learning all tasks) A high ACC indicates the algorithm successfully balances learning new tasks with retaining old knowledge.

Forgetting Measure quantifies catastrophic forgetting by calculating the average drop in performance for each task from its peak accuracy to its final accuracy. Formally, for task k, forgetting is Fk = max_{j ∈ {1,...,T}} (Ak,j) - Ak,T, where Ak,j is accuracy on task k after learning task j. The overall FM is the average of Fk over all tasks. A low FM (near 0) indicates strong stability and minimal forgetting, which is a core goal in continual learning.

Backward Transfer measures the influence of learning new tasks on past ones. It is calculated similarly to the Forgetting Measure but can be positive or negative.

• Positive BWT: Learning a new task improves performance on a past task (e.g., discovering better features).
• Negative BWT: Learning a new task harms past task performance (catastrophic forgetting). On Split CIFAR, algorithms aim for BWT close to zero or slightly positive, minimizing negative interference. It is computed as: BWT = (1/(T-1)) * Σ_{i=1}^{T-1} (Ai,T - Ai,i).

Forward Transfer evaluates how knowledge from previous tasks improves learning on a new task. It is measured by comparing the accuracy on a new task j at the time it is first learned (Aj,j) against the accuracy of a model trained on that task in isolation (from scratch). A positive FWT indicates that prior learning provides a beneficial inductive bias, speeding up learning or improving final performance on the new task. This is a key indicator of positive knowledge accumulation in a continual system.

Learning Curve Area is an online performance metric, particularly relevant for online continual learning variants of Split CIFAR. Instead of only evaluating after all training, LCA computes the area under the learning curve, where the x-axis is the number of training samples (or tasks) seen and the y-axis is the accuracy on a held-out test set for all tasks seen so far. This metric rewards algorithms that learn quickly and maintain high performance throughout the entire stream, not just at the endpoint.

These are critical practical metrics reported alongside accuracy-based scores.

• Parameter Growth: Measures the increase in the total number of trainable parameters over the task sequence. Architectural methods like Progressive Networks have high growth; regularization methods like EWC have zero growth.
• Memory Buffer Size: For replay-based methods (e.g., Experience Replay), the size of the episodic memory buffer (e.g., 2000 samples total for 10 tasks) is a key hyperparameter. Efficiency is the accuracy achieved per unit of memory stored.
• Compute/Time: The total training time or FLOPs required to complete the benchmark. This assesses the algorithmic overhead of the continual learning mechanism.

Continual learning is the overarching machine learning paradigm where a model learns sequentially from a non-stationary stream of data or tasks. Its primary objective is to accumulate knowledge over time without catastrophic forgetting of previously learned information. This is the fundamental problem that benchmarks like Split CIFAR are designed to measure.

• Core Challenge: The stability-plasticity dilemma—balancing the retention of old knowledge (stability) with the acquisition of new knowledge (plasticity).
• Key Scenarios: Includes task-incremental, class-incremental, and domain-incremental learning, which Split CIFAR typically evaluates.

Catastrophic forgetting is the primary failure mode in continual learning, where a neural network abruptly and drastically loses performance on previously learned tasks after being trained on new data. This occurs due to interference as new gradients overwrite weights critical for old tasks.

• Mechanism: Unconstrained optimization for a new task shifts the model's parameters away from the optimal region for prior tasks.
• Benchmark Role: The central metric for algorithms evaluated on Split CIFAR is their ability to minimize this effect while maintaining accuracy on new classes.

Experience Replay (ER) is a foundational rehearsal-based technique to mitigate forgetting. It maintains a fixed-size replay buffer of past training examples. During training on a new task (like a new Split CIFAR subset), the model interleaves learning from new data with rehearsal on stored old examples.

• Core Function: Provides direct exposure to old data distributions, preventing representational drift.
• Variants: Includes Dark Experience Replay (DER), which stores model logits for stronger consistency regularization, and Generative Replay, which uses a generative model to produce synthetic past data.

Elastic Weight Consolidation (EWC) is a seminal regularization-based continual learning algorithm. It estimates the importance of each network parameter for previous tasks (using the Fisher information matrix) and adds a quadratic penalty to the loss function. This penalty slows down learning on important parameters, effectively "consolidating" old knowledge.

• Mechanism: Applies a soft, diagonal constraint to the parameter space based on estimated importance.
• Use Case: A common baseline algorithm for benchmarks like Split CIFAR, representing the parameter-importance family of solutions.

Gradient Episodic Memory (GEM) is an optimization-based algorithm that stores past examples in an episodic memory. Instead of replaying them, it uses them to define constraints. When computing the gradient for a new mini-batch, GEM projects it to the closest direction that does not increase the loss on the memory examples.

• Core Idea: Ensures positive backward transfer or at least minimizes negative transfer by constraining gradient updates.
• Advantage: Provides a theoretical guarantee of non-increasing loss on past tasks, a strong property for sequential learning scenarios like Split CIFAR.

iCaRL is a pioneering algorithm specifically designed for class-incremental learning, the exact setting of Split CIFAR. It combines several key strategies:

• Exemplar Management: Stores a subset of real images (prototypes) for each old class.
• Nearest-Mean Classification: At inference, classifies based on the smallest distance to the mean feature vector of each class's exemplars.
• Knowledge Distillation: Uses a distillation loss to preserve the old model's responses when learning new classes.

It established a strong baseline for the challenging class-incremental scenario without task identifiers.