logo Mr.Robota 具身智能学习手册
/
Chapter 16: Damo Institute & Datawhale Online Team Learning

Python script entry point

These scripts supplement the Notebook, rather than replacing it. The Notebook is used to observe images, actions, and diagnostic metrics cell by cell; the scripts are used for batch evaluation, replaying dataset, and generating reviewable videos. The scripts do not include the dataset, model weights, or the complete mujoco_pnp project.

The multi-bench tutorial also provides two lightweight entries:

  • convert_discoverse_to_lerobot.py: Convert the two-channel video, 7-dimensional state, and 7-dimensional actions of DISCOVERSE AIRBOT into LeRobot data, and output a list of conversions per round;
  • robocasa_match16/sitecustomize.py: Register a fixed RoboCasa365 16 task comparison panel during operation, without modifying the upstream source code of RoboCasa.

Preparations before Running

Set the path in the topic catalog:

bashcd /path/to/every-embodied/16-专题组队学习/04-AMD-ROCm策略复刻专题
export PROJECT_ROOT=/path/to/mujoco_pnp
export DATA_ROOT=/path/to/datasets/every_embodied
export MODEL_ROOT=/path/to/checkpoints/every_embodied
export OUTPUT_ROOT=/path/to/outputs/every_embodied

PROJECT_ROOT must include at least asset/example_scene_y2.xml and mujoco_env/y_env2.py. The file omy_pnp_language/meta/info.json is searched by default under DATA_ROOT; you can also directly set DATASET_ROOT. For the source, size, and external placement of the upstream project, see …/external/README.md.

PyTorch in ROCm still uses DEVICE=cuda. If you are just troubleshooting path and import issues, you can explicitly set DEVICE=cpu, but this does not indicate the actual GPU performance of the model.

For video output, it is preferred to use H.264 encoding with imageio and imageio-ffmpeg; when only opencv-python is available in the environment, the script will automatically fall back to the MP4 encoder that OpenCV can use. It is recommended to install additional packages when the browser plays smoothly:

bashpip install imageio imageio-ffmpeg

Batch Closed Loop Evaluation

run_closed_loop.py directly reads the checkpoint and puts the policy back into SimpleEnv2. It retrieves dual-camera images, 6-dimensional joint states, and language commands at 20 Hz, and then outputs a 7-dimensional action. It also records simultaneously:

  • legacy_success: Old geometric success conditions in the upstream environment;
  • physical_success: Strict conditions adding lift duration, placement height, cup upright position, plate displacement, gripper release, end lift, and continuous stable frames on the old conditions;
  • review fields such as xy_dist, max_target_lift, and upright_cos;
  • H.264/yuv420p rollout videos for each seed (fallback to OpenCV when the encoder is unavailable) and results.jsonl.

Example:

bashexport POLICY_TYPE=smolvla
export MODEL_RUN_DIR="$MODEL_ROOT/smolvla_weighted_000500"
export EVAL_SEEDS=1000,1001,1002,1003
export RENDER=0
python code/run_closed_loop.py

POLICY_TYPE supports act, smolvla, and pi0. If the model directory is already pretrained_model, you can use:

bashexport POLICY_PATH=/path/to/pretrained_model
python code/run_closed_loop.py

The default closed loop does not open a 3D window, making it suitable for Xvfb or windowless servers; set RENDER=1 and prepare DISPLAY first. In formal reports, do not use only the four default seeds, but at least expand to 20–30 held-out seeds, and count them separately by red cups, blue cups, and physical successes.

SmolVLA Quick Access

bashpython code/task11_smolvla_eval.py

It simply sets POLICY_TYPE=smolvla and four default seeds, then calls run_closed_loop.py. If the model path is different, MODEL_RUN_DIR should be explicitly set first; it will not execute code through fixed Notebook cell numbers, nor will it write to /root/.Xauthority on the server.

Dataset Replay

First, list the episode:

bashpython code/replay_dataset.py --list

Export the agent or wrist view again:

bashpython code/replay_dataset.py --episode 0 --view agent
python code/replay_dataset.py --episode 3 --view wrist --fps 20

Default output is saved to $OUTPUT_ROOT/replay_ep{episode}_{view}.mp4. The main perspective of the dataset is prioritized. Read observation.image, while being compatible with the old agent_image naming; read observation.wrist_image from the wrist perspective. If the path is not DATA_ROOT/omy_pnp_language, set:

bashexport DATASET_ROOT=/path/to/omy_pnp_language

MuJoCo Introduction Demo

The small ball Demo is self-contained:

bashpython code/demo/demo_ball.py

The robotic arm Demo requires an upstream scenario:

bashpython code/demo/demo_arm.py

Both videos will be saved as outputs/demo_videos. It is preferred to use the H.264/yuv420p format that is easier to decode in browsers. The Demo only verifies rendering, scenes, and video links, and is not equivalent to a closed loop success rate for policy training.

Common Restrictions

  • The lightweight amd-rocm branch does not submit teaching data, checkpoints, Hugging Face cache, or the complete third_party/;
  • The script must be compatible with the current upstream mujoco_pnp and LeRobot versions. When there is a version mismatch, first check the path and environment audit in Notebook 01;
  • When there is no PROJECT_ROOT, dataset metadata, or model weights, the script will explicitly report the missing paths, and it will not access absolute directories on the author’s machine;
  • physical_success is a strict protocol in the tutorial. Do not consider the four seed smoke results of the script as sufficient generalization conclusions.