Success Weighted by Path Length (SPL)

SPL is a composite metric that balances two critical factors: task success and path efficiency. It is defined as: SPL = (1/N) Σ_i^N S_i * (l_i / max(p_i, l_i)).

• S_i is a binary success indicator (1 for success, 0 for failure).
• l_i is the optimal shortest path length.
• p_i is the actual path length taken by the agent.
• N is the total number of evaluation episodes.

This formula ensures an agent that succeeds but takes a meandering route receives a lower score than one that follows a near-optimal path.

The core innovation of SPL is its path-length penalty. The term l_i / max(p_i, l_i) acts as a penalty coefficient applied to the success indicator.

• If an agent fails (S_i=0), the SPL contribution for that episode is zero.
• If an agent succeeds (S_i=1), its score is reduced proportionally to how much longer its path (p_i) was than the optimal path (l_i).
• An agent taking the exact optimal path (p_i = l_i) achieves the maximum per-episode score of 1.
• An agent taking a longer path (p_i > l_i) receives a score less than 1, e.g., a path twice as long yields a score of 0.5 for that episode.

SPL was introduced as part of the Habitat and AI2-THOR embodied AI challenge suites to provide a standardized, rigorous metric for visual navigation tasks. It addresses the limitation of pure success rate, which does not account for efficiency.

It is the standard metric for evaluating agents in benchmarks like:

• PointNav: Navigate to specified coordinates.
• ObjectNav: Navigate to an instance of a specified object category.

Its design makes it highly relevant for sim-to-real transfer, where evaluating both reliability and the quality of behavior is essential before physical deployment.

A critical prerequisite for calculating SPL is knowledge of the optimal shortest path length (l_i). This is typically computed using privileged information not available to the agent, such as a full map of the environment.

• In simulation, l_i is computed using graph-based planners (e.g., Dijkstra's algorithm) on a navigation mesh.
• This establishes a performance ceiling; an agent cannot score higher than an oracle with perfect knowledge of the shortest path.
• The reliance on a ground-truth optimal path makes SPL a fair metric for comparison across different algorithms, as all are measured against the same theoretical optimum.

SPL produces a single scalar value between 0 and 1 that is easily interpretable.

• SPL = 0: The agent failed every episode or, in vanishingly rare cases, succeeded but took infinitely long paths.
• SPL = 1: The agent succeeded in every episode and took the exact optimal shortest path in each one (the ideal performance).
• 0 < SPL < 1: Represents the average penalized success across all trials. An SPL of 0.7 indicates a high level of both reliable and efficient navigation.

This bounded scale allows for clear, at-a-glance comparison of different policies or sim-to-real transfer methods.

SPL was designed to correct for a key flaw in using success rate alone. Consider two agents:

• Agent A: Succeeds 100% of the time but consistently wanders, doubling the necessary path length.
• Agent B: Succeeds 80% of the time but always takes the optimal path when it succeeds.

A success rate metric ranks Agent A (100%) above Agent B (80%). However, SPL provides a more nuanced view:

• Agent A SPL: 1.0 * (l / 2l) = 0.5 (averaged over episodes).
• Agent B SPL: 0.8 * (l / l) = 0.8.

SPL correctly identifies Agent B as providing more efficient and reliable navigation on average, making it a superior metric for practical deployment.

SPL is essential for A/B testing policies trained in simulation before costly real-world deployment. It provides a single, interpretable number that balances task success with motion efficiency.

• Comparative Analysis: Engineers use SPL to select the most efficient policy from a cohort trained with different Reinforcement Learning (RL) algorithms or domain randomization parameters.
• Robustness Proxy: A high SPL across varied simulated conditions suggests a policy may be robust to distribution shift when transferred to reality.
• Performance Tracking: Monitors improvement across training iterations in simulation, where sample efficiency is critical.

For Autonomous Mobile Robots (AMRs) in warehouses and logistics, SPL translates directly to throughput and cost-per-pick. Optimizing for SPL means minimizing travel distance for successful item retrieval.

• Use Case: Evaluating navigation stacks for goods-to-person picking systems.
• Business Metric: Lower path length for success reduces wear on motors, saves energy, and allows more tasks per charge cycle.
• Integration: Combined with Success Rate to create service-level agreements for robotic fleet performance.

In virtual reality and metaverse applications, SPL evaluates how naturally and efficiently AI-controlled avatars navigate digital spaces to assist users or complete tasks.

• Application: Testing NPC navigation in training simulations or virtual customer service environments.
• User Experience: Efficient paths (high SPL) prevent avatars from appearing confused or obstructive, enhancing immersion.
• Scalability Testing: Used to stress-test navigation systems under load with hundreds of simultaneous agents.

In safety validation for autonomous systems, SPL helps identify policies that are not just successful but predictable and efficient. Erratic, long paths may indicate latent instability.

• Risk Indicator: A policy with high Success Rate but low SPL may take circuitous, unpredictable routes, posing a safety risk in shared human-robot spaces.
• Failure Mode Analysis: Used in simulation-based testing to flag policies that succeed but exhibit inefficient navigation patterns before real-world deployment.
• Complement to Constraints: Evaluated alongside metrics for collision rate and social compliance.

Success Rate is the most fundamental navigation metric, measuring the percentage of trials where an agent reaches its goal. It is the primary component of SPL, but lacks the efficiency penalty. For example, a policy that succeeds 90 out of 100 times has a 90% success rate. However, it does not account for whether the agent took a meandering, inefficient path to achieve that success.

The Path Length Ratio is the efficiency component of SPL. It is calculated as the optimal path length divided by the agent's actual path length. This ratio penalizes unnecessarily long trajectories.

• A ratio of 1.0 indicates the agent followed the perfect shortest path.
• A ratio of 0.5 means the agent traveled twice the optimal distance.
• A ratio of 0.0 is assigned if the agent fails, as the actual path length is considered infinite. SPL multiplies the binary success indicator by this ratio.

A Normalized Score is a performance metric scaled relative to a baseline policy to enable comparison across different tasks or environments. SPL is inherently normalized between 0 and 1. Other common normalized metrics in reinforcement learning include:

• Score normalized by a random agent's performance.
• Score normalized by an expert's or oracle's performance. This scaling is critical for benchmark suites where raw reward values are not directly comparable.

Cumulative Reward, or return, is the sum of all rewards an agent receives during an episode in reinforcement learning. While SPL is a task-specific metric for navigation, cumulative reward is the general objective function the agent is trained to maximize. In navigation tasks, the reward function is often sparse (e.g., +1 for success, 0 otherwise), making cumulative reward equivalent to success rate. SPL provides a more nuanced evaluation by incorporating path efficiency, which a sparse reward does not capture.

A Benchmark Suite is a standardized collection of tasks, environments, and evaluation protocols for systematic algorithm comparison. SPL is a cornerstone metric in major robotics and embodied AI benchmarks, including:

• AI2-THOR (for interactive navigation).
• Habitat and Gibson (for photorealistic navigation).
• RoboTHOR (for sim-to-real transfer in navigation). These suites mandate the use of SPL alongside other metrics (e.g., success rate, distance to goal) to ensure comprehensive and reproducible evaluation.

An Evaluation Protocol is the rigorous, predefined procedure for testing and scoring an algorithm. For SPL, a strict protocol is essential for reproducibility and includes:

• Defining the optimal path length (often pre-computed via a planner like A* on a known map).
• Specifying the number of episodes and starting conditions (e.g., random start/goal pairs).
• Setting termination conditions (time limit, collision, success).
• Calculating SPL across all episodes and reporting the mean and standard error. Adherence to this protocol allows for direct comparison between research papers.