Index of Embodied AI Data and Evaluation Projects

This page includes data, tasks, and evaluation tools that can complement this topic. When selecting a project, you should first check the robot body, action space, task protocol, data permissions, and downloadable resources, and then decide whether to combine it with current experiments.

RoboBenchMart

• Project address: emb-ai/RoboBenchMart
• Appropriate use: for retrieving robot manipulation benchmarks, task definitions, dataset, and related resources.
• Usage advice: use it as a reference navigation entry; do not directly compare the success rates of different projects. Robots, initial states, maximum steps, and success criteria may vary across projects.

Open X-Embodiment

• Project address: Open X-Embodiment
• Appropriate use: To understand how cross-robot data can be uniformly observed, actions, and task descriptions.
• Usage recommendations: Focus on checking ontology mapping, action dimensions, normalization statistics, and data permissions.

LeRobot Dataset

• Project address: Hugging Face LeRobot
• Suitable for: Organizing rounds, images, states, actions, and task texts in a unified directory, and integrating with training and visualization tools.
• Usage advice: When converting existing data, check the frame rate, timestamp, feature shape, and round boundaries item by item.

Checklist for Selecting a Baseline

1. Whether the task covers the generalization aspects that are truly important to the project.
2. Whether a fixed initial state, evaluation script, and success criteria are provided.
3. Whether there is data and policy input/output compatible with the target robot.
4. Whether computing resources, simulation assets, and model weights are available.
5. Whether the license allows use in academic, research, or commercial scenarios.

After determining the baseline, specify the task subset, number of iterations, maximum steps, and aggregation method in the experiment record, so that different models can be compared under the same protocol.