Fairness in Reinforcement Learning

The foundational mathematical framework for Fair RL. A CMDP extends the standard MDP by adding cost functions and cumulative cost constraints that the agent must satisfy while maximizing reward. The agent learns a policy π that maximizes expected return subject to bounds on expected auxiliary costs, such as ensuring a minimum service level for all demographic groups. This transforms fairness from a post-hoc evaluation into a hard optimization constraint.

A critical distinction unique to sequential settings. Instantaneous fairness requires equitable decisions at every timestep, while long-term fairness allows temporary disparities if they correct systemic imbalances over a horizon. For example, a loan agent might temporarily approve fewer applications from a historically advantaged group to equalize aggregate approval rates over a year. This trade-off is formalized through temporal fairness constraints on the cumulative sum of group-specific outcomes.

Fair RL is inherently a multi-objective problem. Agents must optimize a Pareto frontier between reward and fairness metrics. Techniques include:

• Scalarization: Combining reward and fairness penalty into a single weighted objective
• Thresholded Lexicographic Ordering: Prioritizing a fairness threshold, then optimizing reward
• Constrained Policy Optimization (CPO) : A trust-region method that guarantees monotonic improvement on reward while satisfying safety constraints at each update step

A technique that modifies the environment's reward signal to incentivize equitable behavior without altering the core RL algorithm. A fairness bonus is added to the extrinsic reward when the agent takes actions that reduce outcome disparities across groups. This approach leverages the agent's inherent reward-maximizing drive, effectively aligning the agent's objective with the designer's ethical goals through potential-based reward shaping that preserves optimal policy invariance.

The agent's observation space must encode sensitive attribute context to make fairness-aware decisions, but this creates a paradox: using protected attributes can enable fairer outcomes, yet their mere presence in a model raises legal and ethical concerns. Architectures address this through disentangled representations that separate group membership from individual merit, allowing the policy network to condition on group context during training while enabling counterfactual reasoning at inference time.

The theoretical guarantee that a fair RL agent's performance loss relative to an unconstrained optimal policy is bounded. Research establishes sublinear regret for fair algorithms, proving that the cost of fairness diminishes over time as the agent collects more data. For instance, a fair bandit algorithm might achieve O(√T) regret while satisfying demographic parity constraints, formally demonstrating that fairness and efficiency are asymptotically compatible in sequential settings.

The foundational mathematical framework for fairness in RL. A CMDP extends the standard MDP by adding auxiliary cost functions and corresponding constraints that the policy must satisfy. Instead of maximizing expected return unconditionally, the agent must keep the expected cumulative cost for each fairness constraint below a specified threshold. This transforms the optimization into a constrained policy optimization problem, typically solved via Lagrangian methods or trust-region approaches like CPO (Constrained Policy Optimization).

A class of objective functions that encode fairness directly into the reward signal by aggregating individual utilities across a population. Common formulations include:

• Utilitarian: Maximizes the sum of all agents' rewards
• Egalitarian (Maximin): Maximizes the minimum reward received by any agent, directly targeting the worst-off
• Nash Social Welfare: Maximizes the product of rewards, balancing efficiency and equity These functions guide the agent toward policies that consider distributive justice rather than pure cumulative return.

Two contrasting design philosophies in RL policy learning:

Fairness Through Ignorance: Deliberately withholding sensitive attributes from the state representation, assuming this prevents discrimination. This approach is fragile—correlated proxy variables in the state can still encode protected information.

Fairness Through Awareness: Explicitly including sensitive attributes in the state and applying fairness constraints on the policy's output distribution. This allows the agent to actively correct for historical disparities rather than ignoring them, enabling counterfactual fairness guarantees.

The choice of the discount factor γ in RL has profound fairness implications. A low γ (e.g., 0.9) makes the agent myopic, prioritizing immediate rewards and potentially exploiting vulnerable groups for short-term gain. A high γ (e.g., 0.999) forces the agent to consider long-horizon consequences, making it harder to justify transient inequities. Researchers have proposed state-dependent discounting and undiscounted average reward formulations to prevent temporal discounting from masking unfair outcomes that accumulate slowly over time.

In Multi-Agent Reinforcement Learning, fairness operates across two dimensions:

• Intra-agent fairness: Ensuring a single agent's policy treats different user groups equitably
• Inter-agent fairness: Ensuring the distribution of resources, tasks, or rewards among cooperating agents is equitable

Techniques include Shapley value credit assignment to fairly attribute team rewards to individual agents, and lexicographic ordering where agents prioritize fairness objectives before efficiency. This is critical in applications like autonomous traffic management and resource allocation.

Evaluating fairness in policies learned from fixed, historical datasets without online interaction. Key challenges include:

• Distributional shift: The learned policy may visit states underrepresented in the data, where fairness properties are unknown
• Hidden confounding: Historical data reflects past biased decisions, making it difficult to distinguish unfairness from legitimate differences
• Importance sampling bias: Re-weighting techniques used in offline evaluation can amplify variance for minority groups

Solutions involve conservative Q-learning with fairness penalties and doubly robust estimation adapted for group-conditional metrics.