Cumulative Reward

The cumulative reward (G_t), also called the return, is the total discounted sum of rewards from timestep t onward: G_t = R_{t+1} + γR_{t+2} + γ²R_{t+3} + ... = Σ_{k=0}^{∞} γ^k R_{t+k+1}. The discount factor (γ), where 0 ≤ γ ≤ 1, determines the present value of future rewards. A γ close to 1 makes the agent far-sighted, while a γ close to 0 makes it short-sighted, focusing on immediate gains. This formulation is the objective that reinforcement learning algorithms, such as policy gradient methods or Q-learning, are explicitly designed to maximize.

In sim-to-real benchmarking, the cumulative reward is the primary quantitative performance metric for evaluating a policy. It provides a single, comparable scalar that summarizes task completion quality.

• Episode Evaluation: The return is calculated over a complete real-world episode or simulated trial.
• Benchmark Comparison: Standardized benchmark suites (e.g., OpenAI Gym's Robotics tasks, Meta's Habitat) use cumulative reward to rank algorithms.
• Success Correlation: For binary success/failure tasks, the reward is often shaped to be high upon success and zero otherwise, making the average return directly correlate with the success rate.

The raw cumulative reward is only meaningful if the underlying reward function is carefully designed. Poor shaping leads to misaligned or exploitable policies.

• Sparse vs. Dense Rewards: Sparse rewards (e.g., +1 for success, 0 otherwise) are simple but make learning difficult. Dense rewards provide incremental feedback (e.g., negative distance to a goal) to guide learning but can lead to unintended behaviors if not designed correctly.
• Sim-to-Real Consistency: The reward function must be computable in both simulation and reality. This often requires using readily measurable signals (e.g., joint angles, object positions) rather than simulated-only properties.
• Normalization: For comparison across tasks, rewards are often normalized against a random policy baseline or an expert demonstration to produce a normalized score.

The stability of the cumulative reward across domain shifts is a direct measure of policy robustness and the size of the sim-to-real gap.

• Distribution Shift Indicator: A significant drop in average real-world return versus simulation return signals a distribution shift in dynamics, visuals, or observation noise.
• Robustness Proxy: Techniques like domain randomization aim to produce policies whose learned return is high across a wide distribution of simulated parameters, promoting out-of-distribution (OOD) generalization.
• Evaluation Protocol: A rigorous evaluation protocol mandates reporting the mean and standard deviation of cumulative reward over many real-world episodes to account for environmental stochasticity.

While fundamental, cumulative reward has limitations that necessitate complementary metrics for a complete evaluation.

• Goal Misalignment: A policy may maximize reward in a way that doesn't fulfill the true intent (reward hacking).
• Lack of Insight: A single scalar doesn't explain how a task was completed. It must be paired with:
  • Success Rate: The binary outcome.
  • Success weighted by Path Length (SPL): For navigation, combines success with efficiency.
  • Qualitative Video Analysis: For diagnosing failure modes.
• Variance: Returns can have high variance, requiring many episodes for a statistically significant measure.

Maximizing cumulative reward is the optimization target for RL algorithms. The sample efficiency—how many environment interactions are needed to achieve a high return—is critical for real-world training where data is costly.

• Simulation Advantage: Training in simulation allows for near-infinite, parallelized episodes to optimize for return, decoupling sample efficiency from real-world cost.
• Algorithms: Model-free algorithms (e.g., PPO, SAC) directly optimize the expected return. Model-based RL uses the reward to guide planning in a learned dynamics model.
• Transfer Objective: The goal of sim-to-real is to achieve a high real-world return with minimal real-world episodes, leveraging cheap simulation samples.

The most intuitive and critical metric for robotics, success rate measures the percentage of trials in which a policy or agent successfully completes its defined task from start to terminal condition. It provides a binary, high-level view of reliability but lacks nuance on how the task was accomplished.

• Primary Use: Final go/no-go decision for policy deployment.
• Limitation: Does not account for efficiency, safety, or path optimality.
• Example: A door-opening policy with a 95% success rate in simulation but an 82% rate on physical hardware clearly indicates the sim-to-real gap.

Sample efficiency quantifies the number of environmental interactions (samples) an algorithm requires to achieve a target performance level. This is the primary economic driver for sim-to-real transfer, as samples in high-fidelity simulation are orders of magnitude cheaper and faster than real-world robot time.

• Key Trade-off: Algorithms with higher sample efficiency can afford more sophisticated domain randomization or training cycles within a fixed compute budget.
• Measurement: Often plotted as a learning curve (cumulative reward vs. number of training steps).
• Real-World Impact: Directly correlates to the cost and time required for policy adaptation on physical hardware.

Policy robustness is the ability of a learned control policy to maintain successful performance despite encountering distribution shift—variations in environmental conditions, sensor noise, or actuator dynamics not explicitly seen during training. It is the ultimate goal of sim-to-real techniques.

• Engineering for Robustness: Achieved through methods like domain randomization and domain-adversarial training.
• Evaluation: Tested via ablation studies that systematically vary parameters like lighting, friction, or payload mass.
• Outcome: A robust policy exhibits high success rate and stable cumulative reward across these varied conditions.

A normalized score rescales a raw performance metric (like cumulative reward) relative to defined baselines, enabling fair comparison across different tasks or environments that have inherently different reward scales. It answers: "How good is this policy compared to a minimal and optimal agent?"

• Calculation: (Score - Random_Score) / (Expert_Score - Random_Score).
• Purpose: Standardizes results in benchmark suites like OpenAI Gym or DeepMind Control Suite.
• Interpretation: A score of 0.0 represents random performance, 1.0 represents expert performance, and >1.0 indicates super-human performance.

The Real-Time Factor (RTF) is a critical infrastructure metric for simulation-based training, defined as the ratio of simulated time to wall-clock time. An RTF > 1.0 means the simulation runs faster than real-time, directly accelerating the data generation for reinforcement learning.

• Impact on Training: An RTF of 1000 allows collecting 1000 seconds of simulated robot experience per second of compute time.
• Trade-offs: Higher visual or physical simulation fidelity often reduces RTF.
• System Design Goal: Maximize RTF while maintaining sufficient fidelity for effective policy transfer.

Out-of-Distribution (OOD) Generalization is the capability of a machine learning model to perform accurately on input data drawn from a statistical distribution significantly different from its training data. The sim-to-real problem is a quintessential OOD generalization challenge.

• Core Problem: The real-world operational domain is always an OOD test for a simulation-trained model.
• Advanced Techniques: Frameworks like Invariant Risk Minimization (IRM) and Distributionally Robust Optimization (DRO) explicitly train models for OOD robustness.
• Evaluation: Requires a dedicated test set with parameters (e.g., object shapes, surface textures) not seen in the training simulation.