Dark Experience Replay (DER)

Unlike standard Experience Replay (ER) which stores only raw input-output pairs (x, y), DER stores a triplet (x, y, z) where z is the model's logit vector output for input x at the time of storage. These stored logits are called 'dark logits' or 'dark knowledge' as they represent the model's internal state and confidence distribution for that example before any subsequent learning occurs.

The core mechanism for preventing forgetting is a consistency loss applied during replay. When a stored triplet (x, y, z) is sampled from the buffer, the model is trained with a combined loss:

• The standard cross-entropy loss with the true label y.
• A mean squared error (MSE) loss between the model's current logit output for x and the stored past logit z. This MSE loss acts as a knowledge distillation signal, forcing the model's internal representation for old data to remain consistent over time, thereby anchoring its behavior.

DER operates with a fixed-size memory buffer, adhering to the constraints of online continual learning. Common strategies include:

• Reservoir Sampling: A probabilistic method that ensures any incoming example has an equal probability of being retained in the buffer over time, providing a representative sample of the data stream.
• Class-Balanced Sampling: In class-incremental scenarios, the buffer may be managed to maintain a roughly equal number of examples per observed class to mitigate bias. During training, mini-batches are constructed by mixing new task data with data (and its dark logits) uniformly sampled from this buffer.

DER provides several key improvements:

• Enhanced Stability: The consistency loss on dark logits provides a stronger, more direct constraint on parameter change than replaying labels alone.
• Label-Efficiency: It can leverage unlabeled past data; if y is unavailable for a stored x, training can proceed using only the consistency loss with the dark logit z.
• Mitigates Bias: By replaying the model's past distribution (logits) rather than just a hard label, it better preserves the original decision boundaries and uncertainties.

DER is fundamentally an online, self-distillation technique. The model continuously distills knowledge from its past self (represented by the frozen dark logits in the buffer) into its current self. This differs from typical offline distillation which uses a separate, static teacher model. The dark logits z act as a dynamic, evolving teacher that tracks the model's state throughout the continual learning journey.

Effective DER implementation requires tuning:

• Buffer Size: The primary constraint, trading off memory cost with forgetting mitigation. Sizes can range from a few hundred to several thousand examples.
• Loss Weighting (alpha): The hyperparameter α balances the cross-entropy loss and the dark logit consistency loss: L_total = L_ce + α * L_mse. A typical starting value is α = 0.5.
• Logit Temperature: Sometimes a temperature parameter T is applied when computing the MSE loss on logits to soften the distributions, similar to standard knowledge distillation.

Experience Replay (ER) is the foundational rehearsal-based technique upon which DER builds. It maintains a fixed-size memory buffer of past training examples (input-output pairs). During training on new tasks, it interleaves these stored 'experiences' with new data, approximating the i.i.d. (independent and identically distributed) data assumption of standard offline training. This simple rehearsal is highly effective at mitigating catastrophic forgetting by preventing the model's parameters from drifting too far from configurations that solved previous tasks.

Gradient Episodic Memory (GEM) is a more constrained replay-based method. It also stores past examples in an episodic memory. However, instead of simply replaying them, GEM uses gradient projection. When computing a gradient update for a new task, it projects this gradient onto a constraint space defined by the gradients calculated on the stored memories. This ensures the new update does not increase the loss on any past task, providing a stronger guarantee against forgetting than naive replay, albeit with higher computational cost per update.

Elastic Weight Consolidation (EWC) is a premier regularization-based method, representing a different philosophical approach from replay. It adds a quadratic penalty term to the loss function during new task training. This penalty constrains the movement of each network parameter based on its estimated importance for previous tasks, derived from the diagonal of the Fisher Information Matrix. Important parameters are 'anchored' with high elasticity. Unlike DER, EWC requires no memory buffer for raw data, instead storing a per-parameter importance vector, but can struggle with a large number of sequential tasks.

Learning without Forgetting (LwF) is a knowledge distillation (KD)-based method that, like DER, avoids storing raw input data. Instead of a buffer, it uses the model's own outputs. For new task data, it passes the input through a frozen copy of the old model to generate 'soft targets' (logits) for the old tasks. A distillation loss between the current model's outputs and these frozen logits for the old task heads is then used alongside the new task loss. This encourages the model to retain its old responses while adapting, but relies on the new data distribution somewhat activating old task knowledge.

Online Class-Incremental Learning (OCIL) defines the most challenging and realistic evaluation scenario for algorithms like DER. In OCIL, a model learns new classes from a non-i.i.d. data stream one example or mini-batch at a time, with no clear task boundaries. The model must adapt online (immediately) and operate under a strict, fixed memory budget for replay. This setting tests an algorithm's ability to balance the stability-plasticity dilemma under severe constraints, making it a key benchmark for assessing practical utility beyond simpler task-incremental settings.