Generative Replay

In robotics, a single agent must learn to perform new tasks (e.g., grasping different objects, navigating new environments) over its operational lifetime. Generative replay enables this by:

• Training a generative model (like a Variational Autoencoder or Generative Adversarial Network) on sensory data (images, proprioception) from each learned task.
• When learning a new skill, the agent interleaves real sensor data with synthetic samples replayed from the generative model of past tasks.
• This prevents catastrophic forgetting of motor policies and visual representations, allowing the robot to maintain a cumulative skill set without retraining from scratch.

Online recommendation models face concept drift as user preferences evolve. A static model becomes stale. Generative replay facilitates continual personalization:

• A generative model learns the distribution of user interaction data (e.g., item embeddings, click sequences) for a given time period or user cohort.
• As new interaction data streams in, the recommendation model is updated. To preserve knowledge of long-term preferences or seasonal trends, synthetic data from past distributions is replayed.
• This balances plasticity (adapting to new trends) with stability (remembering enduring user tastes), improving long-term engagement metrics.

Hospitals periodically acquire new imaging devices or encounter novel disease subtypes. Retraining a diagnostic model on all historical data is often infeasible due to data privacy regulations (e.g., HIPAA, GDPR). Generative replay offers a compliant solution:

• A differentially private generative model is trained on a legacy dataset of medical images (e.g., X-rays for pneumonia).
• When the model is updated with new data (e.g., images from a new scanner), the private generative model replays synthetic versions of the old data.
• The diagnostic model learns from the new real data and the old synthetic data, maintaining performance on previous diagnostic tasks without directly accessing or storing the original sensitive patient records.

A self-driving car's perception system must recognize objects under countless environmental conditions (weather, lighting, geography). Training on all conditions simultaneously is impossible. Generative replay enables incremental domain adaptation:

• After training in one domain (e.g., sunny California), a generative model captures the visual style and object distribution.
• When deployed to a new domain (e.g., snowy Germany), the model learns from new real snow data. To avoid forgetting how to recognize objects in sun, it is also trained on generated sunny scenes.
• This process can be repeated for each new geographic or climatic region, building a robust, globally capable perception system that adapts locally without forgetting previous domains.

In federated learning, models are trained across decentralized edge devices (e.g., smartphones) without centralizing raw data. Generative replay enables continual federated learning where data on each device is non-IID and changes over time.

• Each client device trains a small generator on its local data distribution for a given task or time window.
• During federated aggregation, clients share generator parameters or generated samples (not raw data) with the server.
• The global model is updated using a mixture of new client data and replayed synthetic data from generators representing past tasks or other clients' distributions. This mitigates catastrophic forgetting across the federation while preserving data privacy.

Language models deployed in production must adapt to new slang, terminology, or writing styles without forgetting standard grammar or common knowledge. Generative replay supports lifelong language learning:

• As a model processes a stream of documents (e.g., news articles, social media posts), a language-based generative model (like a smaller GPT) is periodically trained on snapshots of the data.
• When fine-tuning the primary model on a new topic or style, text samples from the generative model of previous distributions are replayed.
• This prevents catastrophic interference in the model's embedding space and syntactic capabilities, allowing it to integrate new linguistic knowledge while retaining core language understanding. It is a key technique for maintaining Large Language Model utility over time.

A core data structure in reinforcement learning and continual learning that stores past experiences (e.g., state, action, reward tuples). It enables:

• Breaking temporal correlations by randomly sampling old data during training.
• Improving sample efficiency by reusing experiences multiple times.
• Mitigating catastrophic forgetting by interleaving old and new data. Implemented as a fixed-capacity circular buffer or via reservoir sampling for non-stationary streams.

A regularization-based continual learning method that addresses catastrophic forgetting without a replay buffer. It operates by:

• Calculating a Fisher information matrix to estimate the importance of each network parameter for previous tasks.
• Applying a quadratic penalty that slows down learning on important weights during training on new tasks.
• Serving as a complementary approach to generative replay, often used when storing raw data is impractical.

A constrained optimization approach to continual learning that uses a small episodic memory of real examples from past tasks. Its mechanism involves:

• Storing a subset of raw data in a fixed-size buffer.
• When computing gradients for a new task, projecting them to avoid increasing the loss on the episodic memory.
• Ensuring positive backward transfer by improving performance on past tasks while learning new ones. Contrasts with generative replay by storing real data instead of generating synthetic samples.

The primary problem that generative replay and related techniques aim to solve. It refers to the tendency of a neural network to abruptly and drastically lose performance on previously learned tasks when trained on new data. Key characteristics:

• Caused by overwriting shared representations in the network's weights.
• Particularly severe in sequential task learning and online learning scenarios.
• Measured by the drop in test accuracy on earlier tasks after training on subsequent ones.

A learned generative model of an environment's dynamics, often used to generate synthetic experience. In contexts like the Dreamer algorithm, it:

• Learns to predict future latent states and rewards from actions.
• Creates a latent imagination where an agent can be trained entirely on generated rollouts.
• Provides a conceptual parallel to generative replay, where a model generates data for training, but focuses on forward prediction for planning rather than past data reconstruction for retention.

An advanced sampling strategy for experience replay buffers that biases selection toward more informative transitions. It enhances learning by:

• Prioritizing experiences with high Temporal-Difference (TD) error, which indicate surprising or not-yet-learned outcomes.
• Using stochastic prioritization with importance sampling to correct bias.
• Demonstrating that how you sample from memory is as critical as what you store, a principle that can extend to sampling from generative models in replay contexts.