Detailed Explanation of the SimplerEnv Benchmarking System
1 Overview
SimplerEnv is a closed loop simulation environment specifically designed for evaluating real robot policies. It is based on the ManiSkill2 physics engine and aims to bridge the gap between “Sim-to-Real” and “Real-to-Sim”.
This chapter will provide a detailed breakdown of the SimplerEnv benchmark. This benchmark not only focuses on the robot’s motion planning capabilities but also evaluates the robustness of the policy under visual changes and physical interference. According to the official defined evaluation protocol, the core test set can be divided into 18 standard evaluation scenarios, covering two major robot manipulation platforms: Google Robot and WidowX.
Figure 1: Schematic diagram of the Simplerenv environment
2 Robot Platform and Task Classification
The task design of SimplerEnv follows the “Visual Matching” principle, which means that the textures, lighting, and object layouts in the simulation environment are designed to mimic real robot datasets (such as the RT-1 dataset and Bridge dataset) as closely as possible.
2.1 Google Robot Task Group (based on RT-1)
The Google Robot task group primarily simulates the interaction of mobile manipulators in kitchen and office environments. These tasks challenge the robot’s full-body control (Base + Arm) as well as its ability to handle articulated objects (such as drawers).
We have categorized the 14 specific evaluation scenarios of Google Robot into the following three core capabilities:
- Grasping & Manipulation: Grasping objects in various poses.
- Navigation & Reaching: Moving to the base and approaching the target.
- Articulated Object Interaction: Manipulation of drawer opening and closing.
Table 1: List of Detailed Evaluation Scenarios for Google Robot
| Core Manipulation Capability | Task ID | Scenario Description | Number of Scenarios | Technical Challenges |
|---|---|---|---|---|
| Object Grasping (Pick Coke Can) | pick_horizontal_coke_can |
Grabbing a horizontally placed Coke can The target object is upside down on the table; the end effector angle must be adjusted for lateral or dive grasping. | 1 | Object pose estimation, grasping planning |
pick_vertical_coke_can |
Grabbing a vertically placed Coke can The target object is upright—this is the standard grasping task. | 1 | Basic grasping success rate | |
pick_standing_coke_can |
Grabbing a standing Coke can (Note: In SimplerEnv, “vertical/standing” is sometimes considered a minor variation or a synonymous test.) | 1 | Robustness testing | |
| General Grasping (Pick Object) | GraspSingleRandomObject |
Grabbing random objects in the scene The objects are no longer limited to Coke cans; they include randomly sampled geometric shapes or everyday items. | 1 | Generalization capability |
| Move Near (Move Near) | MoveNearGoogleBakedTex |
Move near the scene object The robot must control the chassis to move the end effector close to the target area without collision. | 1 | Whole-body control |
| Open Drawer (Open Drawer) | open_top_drawer |
Open the upper drawer | 1 | Contact force control, kinematic constraints |
open_middle_drawer |
Open the middle drawer | 1 | Different operation heights | |
open_bottom_drawer |
Open the lower drawer | 1 | Low-position manipulation planning | |
| Close Drawer (Close Drawer) | close_top_drawer |
Close the upper drawer | 1 | Force direction control |
close_middle_drawer |
Close the middle drawer | 1 | ||
close_bottom_drawer |
Close the lower drawer | 1 | ||
| Place in Drawer (Place in Drawer) | place_in_closed_top_drawer |
Place into the closed upper drawer A composite task: open the drawer first -> place the object -> (optional) close the drawer. | 1 | Long-horizon task, multi-stage planning |
place_in_closed_middle_ |
Place into the closed middle drawer | 1 | ||
place_in_closed_bottom_ |
Place into the closed lower drawer | 1 | ||
| Total | 14 |
2.2 WidowX Robot Task Group (based on Bridge Data V2)
The WidowX task group simulates a tabletop manipulation robot with a fixed base. The data for this set of tasks comes from Bridge Data V2, focusing on precise table manipulation, stacking, and simple object placement. The environment background typically includes visual diversity due to different laboratory conditions.
Table 4-2: List of Detailed Evaluation Scenarios for WidowX Robot
| Core Task Capability (Capability) | Task ID (Task ID) | Scenario Description (Scenario Description) | Number of Scenarios | Technical Challenges Analysis |
|---|---|---|---|---|
| Precise Placement (Pick & Place) | PutSpoonOnTableCloth |
Placing a spoon on a towel requires identifying the area of the deformable object (towel) and precisely positioning the rigid object (spoon). | 1 | Target area recognition; fine manipulation |
PutCarrotOnPlate |
Placing a carrot on a plate is a typical “object-container” interaction task. | 1 | Depth perception; obstacle avoidance | |
| Object Stacking (Stacking) | StackGreenCubeOnYellow |
Stacking green blocks on yellow blocks is a classic block stacking task requiring extremely high vertical alignment accuracy. | 1 | Precision control; physical simulation of object stability |
| Container Interaction (Container Interaction) | PutEggplantInBasket |
Placing eggplant in a basket involves interaction between an irregular-shaped object (eggplant) and a deep container (basket). | 1 | grasping point selection; preventing falling |
| Subtotal | 4 |
3 Expansion and Variant Aggregation of Evaluation Systems
Although 18 standard scenarios are listed above, readers must understand that the core value of SimplerEnv lies in its breadth and robustness of evaluation. In the actual Benchmark Code, each standard scenario includes a complex “Variant Aggregation” mechanism.
3.1 What is variant aggregation?
In SimplerEnv, a single “task” (such as pick_coke_can) is not static. To accurately evaluate the robot’s generalization ability, the system randomizes the following parameters every time the environment is reset, resulting in hundreds or thousands of actual operating states:
- Visual Variants:
- Background Texture: Change the tablecloth color and floor material.
- Lighting Conditions: Alter the light source intensity and shadow direction.
- Distractors: Generate random objects unrelated to the task in the working space to test the visual model’s attention mechanism.
- Physical Variants:
- Initial Object Position: Randomize the
coordinates and rotation angle of the target object within a specified distribution range. - Camera Pose: Apply minor perturbations to the camera view to simulate errors in camera calibration in real-world scenarios.
- Initial Object Position: Randomize the
3.2 Evaluation Metrics
For the above 18 scenarios, SimplerEnv typically uses the following metrics for quantitative evaluation:
- Success Rate: The percentage of times the task objective is completed successfully during
episodes. It is generally recommended to use for statistical significance. - Average Steps: The average number of control steps required to complete the task, used to measure the efficiency of the policy.
- Sim-to-Real Correlation: This is the most unique metric of SimplerEnv. It calculates the correlation between the success rank in the simulation environment and the policy rank in real-world experiments (e.g., Pearson or Spearman correlation coefficient). A good simulation benchmark should be able to accurately predict which policy performs better on a real robot.
4 Summary of Chapter 4
SimplerEnv defines 18 basic evaluation scenarios to construct a standard test set covering mobile manipulation (Google Robot) and desktop manipulation (WidowX). These scenarios range from simple picking tasks to complex long-distance joint object manipulation. Combined with its underlying visual and physical randomization mechanisms, this benchmark provides a strict and reproducible “examination room” for general robot policies in embodied AI.