iCaRL (Incremental Classifier and Representation Learning)

iCaRL maintains a small, fixed-size episodic memory of real data samples (exemplars) from previously learned classes. This memory is managed using a herding algorithm, which selects a representative subset of images that best approximates the class mean in the feature space. The bounded size creates a fundamental plasticity-stability trade-off, forcing the algorithm to be selective about what past knowledge to preserve.

At inference, iCaRL does not use a standard linear classifier. Instead, it employs a nearest-mean-of-exemplars rule. For each class, a prototype vector is computed as the mean feature vector of all its stored exemplars. A new sample is classified by comparing its extracted features to all class prototypes using Euclidean distance and assigning the label of the closest prototype. This non-parametric rule is inherently adaptable to new classes without retraining the final classification layer.

The core mechanism for preventing catastrophic forgetting is distillation for forgetting. When learning a new set of classes, the model's predictions on new data are combined with its own predictions (from the previous model state) on the same data. A knowledge distillation loss (typically Kullback-Leibler divergence) penalizes deviations from the old model's outputs, effectively encouraging the new model to retain its original behavior on the feature representations relevant to old tasks.

iCaRL's training objective is a weighted sum of two losses:

• A standard cross-entropy loss on the new classes (using both new data and exemplars from the memory).
• The knowledge distillation loss applied to all classes (old and new) using exemplars from the memory. This dual objective ensures the model simultaneously acquires new knowledge (plasticity) and consolidates old knowledge (stability). The feature extractor is updated end-to-end under this combined signal.

Unlike methods that freeze the feature extractor, iCaRL updates the deep representation (the convolutional backbone) continuously. The distillation loss regularizes updates to this shared representation, allowing it to evolve for new tasks while remaining useful for old ones. This enables forward transfer—where learning new classes can sometimes improve performance on old ones—and is key to long-term scalability.

iCaRL operates in a class-incremental learning setting, which is task-agnostic. The model is not provided with a task identifier at test time and must distinguish between all classes seen so far. The nearest-mean-of-exemplars classifier naturally supports this, as the set of class prototypes is simply expanded as new tasks arrive. This makes the algorithm suitable for real-world deployment where task boundaries are unclear.

Knowledge distillation is a model compression and transfer learning technique where a smaller student model is trained to mimic the behavior of a larger teacher model. In iCaRL, this principle is applied internally: the model from the previous learning step acts as the teacher, providing soft targets to regularize training on new data, thereby preserving old knowledge. The loss function typically uses Kullback-Leibler (KL) Divergence to match probability distributions.

• Core Mechanism: Transfers 'dark knowledge' from logits.
• iCaRL Role: Prevents catastrophic forgetting by distilling old model's predictions.

Catastrophic forgetting, or catastrophic interference, is the tendency of a neural network to abruptly and drastically lose performance on previously learned tasks when trained on new data. It is the central problem iCaRL is designed to solve. This occurs due to the overwriting of shared weights during gradient-based optimization on a non-stationary data stream.

• Cause: Unconstrained parameter updates for new tasks.
• iCaRL's Solution: Combines knowledge distillation loss with exemplar rehearsal to constrain updates and consolidate old knowledge.

Experience replay is a continual learning mechanism where a subset of past training examples is stored in a memory buffer and replayed during new learning phases. iCaRL implements a bounded exemplar memory, storing a fixed number of representative images per old class using a herding algorithm. This provides direct exposure to old data.

• Memory Management: Uses herding to select representative exemplars.
• Function: Provides real data for rehearsal alongside distillation, grounding the regularization signal.

Class-incremental learning (CIL) is the specific continual learning scenario addressed by iCaRL. The model must learn new classes sequentially over time while maintaining performance on all previously seen classes, without receiving task identifiers at inference. This is more challenging than task-incremental learning, as the model must autonomously discriminate between all classes.

• Key Challenge: No task-ID provided during prediction.
• iCaRL's Classifier: Employs a nearest-mean-of-exemplars rule in the feature space, which is inherently task-agnostic.

This is the novel classification rule introduced by iCaRL for the class-incremental setting. Instead of using a standard linear output layer (which biases towards newly learned classes), iCaRL classifies based on distance in feature space. For each class, a prototype vector is computed as the mean feature vector of its stored exemplars. At inference, a sample is assigned to the class with the closest prototype.

• Advantage: Naturally accommodates new classes without retraining the classifier head.
• Requirement: Depends on a stable, high-quality feature representation learned via distillation.

Learning without Forgetting is a seminal continual learning algorithm that, like iCaRL, uses knowledge distillation to preserve old knowledge. However, LwF is exemplar-free; it relies solely on distilling the model's own responses on new data to approximate its old behavior. iCaRL extends this concept by adding an exemplar memory, which provides a more stable and accurate distillation target and enables the nearest-mean classifier.

• Comparison: LwF is a pure distillation approach, while iCaRL is a hybrid (distillation + replay) method.
• Trade-off: iCaRL typically achieves higher accuracy at the cost of maintaining a memory buffer.