Gradient Projection

The fundamental mechanism of Gradient Projection is to treat the gradient vector for a new task as an optimization variable that must satisfy a set of inequality constraints. These constraints are defined by the inner product between the proposed gradient and the gradient vectors computed on a small episodic memory of past tasks. The update is projected onto the closest valid direction that does not violate these constraints, mathematically ensuring the loss on previous tasks does not increase.

The projection operation is solved as a Quadratic Programming (QP) problem. Given a proposed gradient g and a set of constraint gradients g_1, g_2, ..., g_t from past tasks, the algorithm finds the new gradient g̃ that minimizes the squared Euclidean distance ||g - g̃||², subject to the constraints g̃ ⋅ g_i ≤ 0 for all past tasks i. This ensures the update direction has a non-positive dot product with past gradients, meaning it does not increase their respective losses.

Gradient Projection relies on a fixed-size episodic memory buffer that stores a subset of raw examples from previous tasks. This buffer is critical because:

• It provides the data needed to compute the constraint gradients for past tasks.
• The quality and diversity of samples in the buffer directly impact the effectiveness of the projection.
• Unlike pure rehearsal, the buffer is used only to compute constraints, not for direct interleaved training, making it a more memory-efficient form of knowledge preservation.

Gradient Projection is the defining component of two major algorithms:

• Gradient Episodic Memory (GEM): The original formulation that solves the QP exactly, guaranteeing no forgetting but at a higher computational cost.
• Averaged GEM (A-GEM): A highly efficient approximation that computes constraints using an average gradient over the entire memory buffer. Instead of satisfying all constraints individually, A-GEM ensures the proposed update does not increase the average loss on the buffer, drastically reducing solve time with minimal performance loss.

Compared to regularization-based methods like EWC that add a soft penalty to the loss, Gradient Projection offers distinct benefits:

• Hard Guarantees: Provides a mathematical guarantee (subject to memory buffer coverage) that loss on past tasks will not increase, whereas penalties only discourage change.
• Dynamic Adaptation: The constraint set updates with each new task, allowing the feasible update space to evolve, unlike static regularization strengths.
• Mitigates Gradient Conflict: Explicitly resolves the directional conflict between the gradients of new and old tasks, rather than just slowing down updates on important weights.

Implementing Gradient Projection introduces specific engineering challenges:

• QP Solver Overhead: The core projection requires solving a QP, which can be a bottleneck. Efficient dual solvers or approximations like A-GEM are essential for scalability.
• Memory Buffer Management: Strategies for sample selection (e.g., reservoir sampling, herding) are crucial to maximize constraint coverage with limited storage.
• Task-Agnostic Inference: Like most rehearsal-based methods, it operates in a task-agnostic setting during inference, as the memory buffer is only used during training to compute constraints.

Gradient Episodic Memory (GEM) is the foundational algorithm that introduced the Gradient Projection technique. It stores a subset of past examples in an episodic memory. During training on a new task, GEM computes the gradient for the new data and projects it onto a constraint space defined by the gradients of the stored past examples. This projection ensures the update does not increase the loss on previous tasks, directly implementing the core idea of Gradient Projection.

Averaged Gradient Episodic Memory (A-GEM) is a computationally efficient variant of GEM. Instead of enforcing constraints against all past tasks individually, A-GEM computes a single average gradient over the episodic memory. The new task's gradient is then projected to satisfy this single, averaged constraint. This reduces the quadratic programming complexity of GEM to a simple inner product, making it more scalable for a large number of tasks, though with a potentially looser guarantee.

The core mathematical operation in the original GEM algorithm is solving a Quadratic Programming (QP) problem. The objective is to find the new gradient update closest to the desired update (for the new task) that satisfies all inequality constraints (loss on past tasks should not increase).

• Objective: Minimize the distance between the projected and desired gradient.
• Constraints: Each past task's gradient defines a half-space the update must lie within.
• Implementation: Often uses libraries like CVXOPT or a dual formulation for efficiency.

Gradient Projection directly addresses the Stability-Plasticity Dilemma, the fundamental trade-off in continual learning.

• Stability: The system's ability to retain knowledge from previous tasks. Gradient Projection enforces stability by constraining updates.
• Plasticity: The system's ability to learn new tasks efficiently. The projection finds the closest permissible update, preserving as much of the new learning signal as possible. The technique explicitly formulates this dilemma as an optimization problem, seeking the optimal balance for a given memory budget.

Backward Transfer (BWT) is a key evaluation metric for continual learning algorithms like those using Gradient Projection. It measures the influence of learning a new task on the performance of previously learned tasks.

• Negative BWT: Indicates catastrophic forgetting (performance on old tasks drops).
• Zero BWT: Indicates perfect stability (old task performance is preserved).
• Positive BWT: Indicates the new task learning actually improved performance on old tasks. Gradient Projection algorithms are explicitly designed to achieve BWT >= 0, preventing negative transfer.

The Episodic Memory Buffer is the fixed-capacity storage component essential for replay-based methods like GEM. Its management is critical for Gradient Projection's effectiveness.

• Content: Stores raw input-output pairs (x, y) from past tasks.
• Sampling Strategy: Determines which examples to retain (e.g., reservoir sampling, herding).
• Size Trade-off: A larger buffer improves constraint accuracy but increases compute and storage. The gradient constraints in Projection methods are only as good as the representativeness of the data in this buffer.