Soft Actor-Critic (SAC)

SAC augments the standard reinforcement learning objective by adding an entropy bonus to the reward function. Instead of solely maximizing cumulative reward, the policy is trained to maximize a trade-off between reward and the entropy of the policy.

• The objective becomes: π* = argmax E[∑ γ^t (R(s,a) + α H(π(·|s)))]
• Entropy (H) measures the randomness of the policy; higher entropy encourages exploration.
• The temperature parameter (α) controls the relative importance of entropy versus reward.
• This prevents premature convergence to suboptimal deterministic policies and produces inherently stochastic behaviors.

SAC is an off-policy algorithm, meaning it learns from data generated by a different policy than the one currently being optimized. It stores transitions (s, a, r, s', done) in a large replay buffer and samples random mini-batches for training.

• Sample efficiency: Each collected transition can be reused many times for gradient updates.
• Decouples data collection from learning: The policy can be updated multiple times per environment step.
• Stabilizes training: Random sampling breaks temporal correlations between consecutive samples, reducing variance in updates.

SAC employs a hybrid actor-critic structure with a specific technique to combat overestimation bias.

• Actor (Policy Network): Outputs the mean and standard deviation of a Gaussian distribution over actions. Actions are sampled using the reparameterization trick to enable gradient backpropagation.
• Critic (Value Networks): SAC uses two separate Q-function networks (twin critics). The minimum Q-value between the two is used for computing the target and policy gradient.
• Clipped Double-Q: This trick mitigates the overestimation bias common in actor-critic methods, where Q-functions systematically overpredict values, leading to policy degradation.

The entropy temperature parameter α is critical but difficult to set manually. SAC can automatically adjust α during training by treating it as a constrained optimization problem.

• A target entropy level H_target is set, typically equal to the negative of the action space dimensionality: -dim(A).
• The temperature is updated via gradient descent to minimize the loss: J(α) = E[-α log π(a|s) - α H_target].
• If the policy's entropy drops below the target, α increases to encourage more exploration. If entropy is too high, α decreases to focus on reward exploitation.
• This removes a sensitive hyperparameter and makes SAC robust across diverse environments without manual tuning.

SAC is purpose-built for continuous action spaces, making it ideal for robotics, autonomous driving, and radio resource management.

• The policy outputs parameters of a Gaussian distribution (mean μ and standard deviation σ) for each action dimension.
• Actions are squashed through a tanh function to bound them within a valid range [-1, 1].
• The log-probability calculation includes a correction term for the tanh transformation, ensuring correct gradient computation.
• This stochastic, bounded output is critical for smooth control tasks like beamforming optimization and power control in wireless networks.

SAC is theoretically grounded in soft policy iteration, a provably convergent algorithm in tabular settings. It extends this to continuous domains using deep neural networks.

• Soft Policy Evaluation: The twin critics are trained to minimize the soft Bellman residual using clipped double-Q targets.
• Soft Policy Improvement: The actor is updated by minimizing the Kullback-Leibler divergence between the policy and the exponential of the Q-function.
• This alternating optimization converges to the optimal maximum-entropy policy, providing strong theoretical guarantees absent in many heuristic deep RL methods.