Multi-Objective Reinforcement Learning (MORL)

Scalarization transforms the multi-objective problem into a single-objective one by aggregating the reward vector. The agent learns a policy for a specific scalarization, defined by a weight vector representing the relative importance of each objective.

• Weighted Sum: The most common method, where the scalar reward is a linear combination: r_scalar = w1*r1 + w2*r2 + ... + wn*rn.
• Key Limitation: A single run finds one point on the Pareto front. To approximate the full front, multiple runs with different weight vectors are required.
• Use Case: Ideal when decision-maker preferences (weights) are known and fixed, such as balancing latency and accuracy in a real-time system.

These methods directly search for a set of non-dominated solutions that approximate the Pareto front. Instead of a single policy, the goal is to learn a Pareto set of policies covering optimal trade-offs.

• Population-Based Algorithms: Extend evolutionary algorithms like NSGA-II to the RL context, maintaining a population of policies that evolve towards the Pareto front.
• Policy Archives: The algorithm maintains an archive of high-performing, diverse policies, updating it based on Pareto dominance relations.
• Use Case: Essential for exploratory phases where the trade-off landscape is unknown, allowing stakeholders to visualize and later choose from a set of optimal compromises.

This approach aims to learn a single, parameterized policy that can be conditioned on a preference vector. The policy π(a|s, w) takes the state s and a preference weight vector w as input, generating actions appropriate for that specific trade-off.

• Generalization Across Preferences: After training, a continuous range of behaviors can be elicited by varying w without retraining, enabling real-time adjustment of agent priorities.
• Architecture: Typically implemented via a neural network with separate input streams or embeddings for the state and preference vector.
• Use Case: Dynamic environments where objectives' importance changes, such as a robot alternating between speed and safety based on context.

These algorithms explicitly learn a finite set of distinct policies, each excelling at a different region of the objective space. The final output is a corpus or portfolio of specialist policies.

• Envelope Method: Learns a set of Q-functions for different scalarizations simultaneously, improving sample efficiency over independent runs.
• Selector Networks: A high-level mechanism may be trained to choose the most appropriate specialist policy for a given situation or preference.
• Use Case: Applications requiring a discrete set of well-understood, reliable modes, such as an autonomous vehicle with distinct 'Eco', 'Balanced', and 'Performance' driving policies.

This formulation treats some objectives as constraints to be satisfied, optimizing the primary objective subject to these limits. It transforms the MORL problem into Constrained RL.

• Objective-to-Constraint: For example, maximize performance (reward) subject to a safety cost being below a threshold.
• Lagrangian Methods: Use dual variables (Lagrange multipliers) to adaptively balance the reward and constraint satisfaction during training.
• Use Case: Safety-critical applications where certain metrics (e.g., error rate, risk exposure) must be kept within absolute bounds, common in finance and healthcare.

MOBO is a sample-efficient framework for optimizing expensive-to-evaluate functions. In MORL, it's used to optimize policy hyperparameters or neural architectures across multiple objectives.

• Surrogate Models: Gaussian Processes model the unknown performance landscape (e.g., latency vs. accuracy) of a policy configuration.
• Acquisition Functions: Guides the search for new configurations by balancing exploration and exploitation on the Pareto front, using metrics like Expected Hypervolume Improvement.
• Use Case: Tuning large language model serving parameters or robot control policies where each training/evaluation cycle is computationally costly.

Pareto optimality is a state in multi-objective optimization where no objective can be improved without worsening at least one other objective. A solution is Pareto optimal if it is not Pareto dominated by any other feasible solution. This concept is central to MORL, as the goal is to discover a set of policies that represent different optimal trade-offs.

• In MORL, a policy is Pareto optimal if no other policy yields higher rewards for all objectives simultaneously.
• The set of all Pareto optimal solutions forms the Pareto front, which visualizes the best possible compromises.

Scalarization is a foundational technique in MORL that transforms a vector-valued reward signal into a single scalar quantity, allowing the use of standard single-objective RL algorithms. This is typically achieved by defining a utility function that aggregates the multiple objectives.

Common methods include:

• Weighted Sum Method: Combines objectives using a fixed weight vector, U = w1*R1 + w2*R2 + ... + wn*Rn.
• Epsilon-Constraint Method: Optimizes one primary objective while treating others as constraints with allowable violation ε.
• A key challenge is that a single scalarization yields only one point on the Pareto front; discovering the full front requires multiple runs with different parameters.

A Multi-Objective Evolutionary Algorithm (MOEA) is a population-based metaheuristic optimization algorithm designed to approximate the Pareto front for problems with multiple, often conflicting, objectives. MOEAs are frequently integrated with or inspire MORL approaches for policy search.

Key algorithms include:

• NSGA-II (Non-dominated Sorting Genetic Algorithm II): Uses non-dominated sorting and crowding distance to maintain diversity.
• MOEA/D (Multi-Objective EA Based on Decomposition): Decomposes the problem into scalar subproblems.
• In MORL, MOEAs can evolve a population of policies, evaluating each based on its vector of cumulative rewards and using Pareto dominance for selection.

The Hypervolume Indicator (or S-metric) is a Pareto-compliant performance metric that measures the volume of the objective space dominated by a set of solutions relative to a predefined reference point. It is the gold standard for quantitatively comparing the quality and coverage of approximate Pareto fronts in MORL.

• A larger hypervolume indicates a better set of solutions, as it implies greater dominance of the objective space.
• It simultaneously rewards convergence (closeness to the true Pareto front) and diversity (spread across the front).
• Calculating hypervolume becomes computationally expensive as the number of objectives increases, a challenge in Many-Objective Optimization (MaOO).

Multi-Criteria Decision Making (MCDM) is a broader field that encompasses methodologies for evaluating and selecting among alternatives based on multiple, often conflicting, criteria. MORL is an automated search technique within MCDM, with the final step often involving a human decision-maker.

• Preference Articulation: The process where a decision-maker's priorities (e.g., weights, goals, reference points) are incorporated to select a single policy from the Pareto front.
• Goal Programming: An MCDM approach where the algorithm minimizes deviation from predefined target levels for each objective.
• MCDM frameworks provide the interface between the MORL algorithm's output (a set of trade-offs) and the final operational policy choice.

Constrained Multi-Objective Optimization involves finding Pareto optimal solutions that also satisfy a set of equality or inequality constraints. In MORL, this translates to learning policies that optimize multiple reward signals while adhering to hard safety, resource, or ethical limits.

• Constraints can be on the state space (e.g., avoid certain regions), action space (e.g., torque limits), or objective space (e.g., cost must be below a threshold).
• Techniques include penalty methods, feasibility-first sorting in algorithms like NSGA-II, and treating constraints as additional objectives.
• This is critical for real-world MORL deployment in areas like robotics and finance, where unsafe or infeasible policies are unacceptable.