Soft Actor-Critic (SAC)

SAC augments the standard RL objective by adding an entropy bonus to the reward function. This explicitly encourages the policy to act as randomly as possible while still succeeding at the task.

• Entropy coefficient (α): Balances exploration vs. exploitation
• Prevents premature convergence to suboptimal deterministic policies
• The policy is incentivized to keep multiple options open, improving robustness to perturbations

SAC uses three networks trained with experience replay for high sample efficiency.

• Actor (Policy): Outputs the mean and standard deviation of a Gaussian distribution over actions
• Twin Soft Q-networks: Two critics are trained independently to estimate the soft Q-value, and the minimum is used to combat overestimation bias
• Target Networks: Slowly updated polyak-averaged copies of the Q-networks for stable learning

Instead of treating the entropy coefficient α as a fixed hyperparameter, SAC can learn α automatically by treating it as a Lagrangian dual variable.

• Formulated as a constrained optimization: minimize policy loss subject to a minimum expected entropy constraint
• α is updated via gradient descent to keep policy entropy above a target value
• Eliminates the need for manual tuning of the exploration-exploitation tradeoff across different environments

SAC uses a stochastic Gaussian policy parameterized by a neural network, which is essential for the entropy framework.

• The actor outputs μ and σ for each action dimension
• Actions are sampled using the reparameterization trick: a = μ + σ · ε, where ε ~ N(0,1)
• This allows gradients to flow through the sampling process, enabling direct policy optimization via backpropagation

SAC modifies the standard Bellman equation by incorporating the entropy of the policy into the target value.

• Soft Q-target: r + γ · (min Q'(s') - α · log π(a'|s'))
• The log-probability term penalizes states where the policy is too deterministic
• This soft backup prevents the value function from collapsing to a single high-reward trajectory, maintaining a richer representation of the state space

SAC achieves state-of-the-art performance on continuous control benchmarks like MuJoCo and DeepMind Control Suite with significantly fewer environment interactions than on-policy methods like PPO.

• Off-policy data reuse via a large replay buffer
• Twin critics reduce positive bias in policy improvement
• Stochasticity provides natural exploration without external noise processes
• Robust to hyperparameter choices due to automatic entropy adjustment

The foundational two-component framework that SAC extends. The actor (policy network) selects actions given a state, while the critic (value network) evaluates how good those actions are. SAC uses twin Q-networks as critics to mitigate overestimation bias, a common failure mode in value-based RL. The actor is trained to maximize both expected Q-value and entropy, while critics are updated via clipped double-Q learning on off-policy data.

The fundamental dilemma SAC addresses through entropy regularization. Pure exploitation greedily selects known high-reward actions, risking convergence to suboptimal policies. Pure exploration gathers information but sacrifices performance. SAC's maximum entropy framework formalizes this balance by adding an entropy bonus to the reward function, incentivizing the policy to remain stochastic and explore proportionally to the uncertainty in each state. The temperature parameter α controls this tradeoff and can be automatically tuned.

A critical component enabling SAC's off-policy learning. Transitions (state, action, reward, next state, done) are stored in a large replay buffer and uniformly sampled during training. This breaks the temporal correlation between consecutive samples, stabilizing neural network training. SAC's off-policy nature means it can learn from data collected by any policy, including historical demonstrations, making it significantly more sample-efficient than on-policy alternatives like PPO.

The primary on-policy alternative to SAC. PPO uses a clipped surrogate objective to constrain policy updates within a trust region, preventing destructively large parameter changes. Key differences from SAC:

• On-policy: requires fresh data for each update, reducing sample efficiency
• Deterministic or stochastic: but without explicit entropy maximization
• Simpler architecture: single critic, no twin Q-networks PPO excels in environments where simulation is cheap and wall-clock speed matters more than sample efficiency.

SAC is a model-free algorithm, meaning it learns directly from interaction without building an explicit model of environment dynamics. This contrasts with model-based RL, which learns or uses a dynamics model for planning. Tradeoffs:

• Model-free (SAC): Higher sample complexity but avoids compounding model errors; better for complex contact-rich tasks where dynamics are hard to model
• Model-based: More sample-efficient but susceptible to model bias; excels in tasks with smooth, predictable dynamics Hybrid approaches combine both for planning in learned latent spaces.

The theoretical foundation distinguishing SAC from standard RL. Instead of maximizing only expected return, SAC maximizes E[Σ r(s,a) + αH(π(·|s))], where H is policy entropy. Benefits include:

• Improved exploration: policy remains stochastic in uncertain regions
• Robustness: learns multiple near-optimal behaviors, adapting better to perturbations
• Graceful failure: doesn't collapse to a single brittle strategy
• Automatic temperature tuning: α can be learned by treating entropy as a constraint, targeting a desired minimum entropy level.