Differentially Private Reinforcement Learning

The central challenge of DP-RL is managing the privacy budget (ε) across a sequential decision process. Unlike supervised learning with a fixed dataset, an RL agent accumulates privacy loss with every interaction. Composition theorems dictate that the total ε scales with the number of timesteps, requiring advanced accounting techniques like the Moments Accountant or Rényi DP to provide tight bounds. Without careful allocation, the budget is exhausted long before the agent converges to an optimal policy, rendering the privacy guarantee meaningless.

In many RL applications, the reward signal itself is sensitive. Consider a healthcare agent learning treatment policies: the reward indicates patient outcomes, which directly reveals health status. DP-RL must add calibrated noise to the observed reward at each step. The sensitivity of the reward function—the maximum change in reward if a single individual's data is altered—determines the noise scale. This is typically bounded by clipping the reward to a known range before applying the Gaussian mechanism.

The agent's observation of the environment state can leak information about the underlying private data used to simulate or model the environment. In model-based DP-RL, the transition dynamics are learned from sensitive trajectories, and every query to the model must be privatized. In model-free DP-RL, the state-action pairs visited during exploration are protected by injecting noise into the policy gradient or Q-value updates. This ensures an adversary observing the agent's behavior cannot infer the specific transitions of any single individual in the training data.

Adapting policy gradient algorithms like REINFORCE or PPO requires privatizing the gradient estimate computed from a batch of trajectories. The standard approach uses DP-SGD adapted for the RL setting:

• Per-trajectory gradient clipping: Bound the L2 norm of the gradient contribution from each individual trajectory to limit sensitivity.
• Gaussian noise injection: Add noise scaled to the clipping threshold and the desired (ε, δ) guarantee.
• Privacy amplification by subsampling: Randomly sample a subset of trajectories for each update to amplify the privacy guarantee, though this introduces additional variance in the policy update.

Offline (Batch) DP-RL operates on a fixed, pre-collected dataset of trajectories. This allows for a finite privacy budget to be allocated once using standard DP-ERM techniques, making privacy accounting simpler. Online DP-RL involves an agent actively interacting with a live environment, where privacy loss accumulates continuously. This requires real-time budget monitoring and often employs a privacy odometer that halts interaction when the budget is exhausted. The online setting is significantly more challenging due to the exploration-exploitation tradeoff under a decaying privacy budget.

In DP Q-Learning, the update rule for the Q-function is privatized. The temporal difference (TD) error, computed from the observed reward and next state, is treated as a query on the sensitive data. Key approaches include:

• Output perturbation: Add noise directly to the computed TD error or the updated Q-value.
• Gradient perturbation: If the Q-function is parameterized by a neural network (DQN), apply DP-SGD to the gradient of the TD loss.
• LDP for multi-agent RL: In decentralized settings, each agent applies Local Differential Privacy to its own observations before sharing Q-value updates with peers, protecting against a curious aggregator.

A critical preprocessing step that bounds the influence of any single trajectory on the policy update. The mechanism:

• Computes the per-example gradient of the loss with respect to each sampled transition.
• Scales the gradient vector if its L2 norm exceeds a fixed clipping threshold.
• Ensures the sensitivity of the training batch is strictly bounded.

Without clipping, outlier rewards or rare state transitions could leak disproportionate information about the environment dynamics, violating privacy guarantees. The clipping threshold directly trades off utility vs. privacy: tighter clipping adds less noise but introduces more bias.

The finite, quantifiable limit on total privacy loss allocated across all queries to the agent's experience replay buffer or environment interactions. In DP-RL:

• Each training epoch or policy update consumes a portion of the budget.
• Once exhausted, further access to sensitive trajectories must be denied to prevent reconstruction.
• Budget allocation strategies must balance exploration efficiency against long-term privacy.

A typical budget might be ε = 8 for training a Q-network on medical treatment sequences, after which the model is frozen and the replay buffer is discarded.

A formal rule quantifying how the total privacy guarantee degrades when multiple DP mechanisms are applied sequentially. In RL, this governs:

• Sequential composition: Each policy gradient step consumes budget additively (or sublinearly under advanced composition).
• Parallel composition: Processing disjoint subsets of the replay buffer does not compound privacy loss.
• Adaptive composition: The agent's choice of actions depends on previous noisy updates, requiring careful accounting.

Understanding composition is critical because an RL agent may perform thousands of updates on overlapping experience data, rapidly consuming the privacy budget if not carefully managed.

The noise injection mechanism most commonly paired with DP-SGD for achieving (ε, δ)-differential privacy in policy gradient methods. It operates by:

• Drawing noise from a Gaussian distribution with zero mean and standard deviation calibrated to the L2 sensitivity of the clipped gradients.
• Providing approximate DP guarantees, where δ represents a small failure probability (typically < 10^-5).
• Offering tighter composition properties than the Laplace Mechanism in high-dimensional parameter spaces.

For a deep Q-network with millions of parameters, the Gaussian mechanism enables practical privacy budgets where the Laplace mechanism would require prohibitive noise levels.