Benchmark Results
Evaluation Protocols
In SEANavBench, candidate algorithms' performance are measured on a suite of episodes, comprising both algorithmic and replay-based pedestrians. We do not provide a single number leaderboard, rather, a full characterization of candidate algorithms across metric subgroups quantifying Safety, Efficiency, and Socialness are reported. Candidate algorithms are free to use either perfect or visual sensing (via simulated RGBD), and this will be used to differentiate posted results.
Metrics
SEANavBench evaluates candidate social navigation algorithms on a number of metrics that help characterize their safety, socialness, and efficiency. They are tabulated below:
SEANavBench Evaluation Metrics
Metric (units) |
Description |
|---|---|
| Completed | true when the robot's final pose is within a specified distance of the goal. The final distance threshold is easily adjustable by the user, but defaults to 1.2m |
| Total Path Length (meters) | geodesic distance of all poses in a robot's path sampled at 1Hz |
| Minimum Distance to Target (meters) | the closest the robot passes to the target position |
| Final Distance to Target (meters) | distance between the last robot position and the target position |
| Time Not Moving | seconds that the robot was not moving. |
| Path Irregularity(radians) | total rotations in the robot's traveled path greater than the total rotations in the search-based path from the starting pose. |
| Path Efficiency | ratio between robot's traveled path and geodesic distance of the search-based path from the starting pose. |
| Robot on Person Personal Distance Violation | number of times a robot approaches a person within the personal distance of the robot. |
| Person on Robot Personal Distance Violation | number of times a person approaches the robot within the personal distance of the robot. |
| Intimate Distance Violation | number of times the robot approached within the intimate distance of a person. |
| Person on Robot Intimate Distance Violation | number of times a person approaches the robot within the intimate distance of the robot. |
| Robot on Person Collision | number of times robot collides with a person. |
| Person on Robot Collision | number of times person collides with a robot. |
| Static Obstacle Collision | number of times the robot collides with a static obstacle. |