Spring Festival Gala Dance Robot Recreation: Complete Practical Record (2026 Version)

This document records the key steps of installing, troubleshooting, and finally making the repository operational. The goal is to help new users avoid common pitfalls as much as possible.

Video tutorial (official website): https://www.datawhale.cn/learn/content/258/6228

Video tutorial (Bilibili): https://www.bilibili.com/video/BV1gBfCBpEEb

Video tutorial supplement: https://www.bilibili.com/video/BV1mjcXzLEGC/?share_source=copy_web&vd_source=cc2570142da6f3f02e6aba37bd88f313

Supplementary document: https://my.feishu.cn/docx/OzdvdvZ3LoSUfSxziNCcFWYFnqa?from=from_copylink

1. Project Objectives

Input: text action description or existing video
Output: robot actions robot_motion*.pkl and visualization (Web/viser/MuJoCo)
Supports multi-robot trajectory (all tracks) and MuJoCo recording for multiple robots

II. Core Process

textPrompt / Video -> PromptHMR -> SMPL-X -> GMR -> Robot Motion

Common commands:

bashpython scripts/generate_video.py --model seedance --action "动作序列：角色向前走四步"
python scripts/extract_pose.py --project data/video_001
python scripts/convert_to_robot.py --project data/video_001 --all-tracks
python scripts/visualize.py --project data/video_001 --robot-viser --robot-all

III. Environment Installation (Recommended)

0) Clone the new machine (do this first)

bashgit clone https://github.com/datawhalechina/every-embodied.git
cd every-embodied/07-机器人操作、运动控制/Locomotion/video2robot
cd third_party
# 3. 克隆 GMR
git clone --depth 1 https://github.com/taeyoun811/GMR.git GMR
# 4. 克隆 PromptHMR
git clone --depth 1 https://github.com/taeyoun811/PromptHMR.git PromptHMR
cd ..

Instructions:

All commands in this section are executed by default in the every-embodied/07-机器人操作、运动控制/Locomotion/video2robot directory.

If you receive a patch delivery package, apply 3 more patches:

bashgit apply patches/main.patch
git -C third_party/PromptHMR apply ../../patches/prompthmr.patch
git -C third_party/GMR apply ../../patches/gmr.patch

Then create the environment with YAML in one click:

bashconda env create -f envs/gmr.yml
conda config --add envs_dirs /root/gpufree-data/conda_envs
# 第一个环境安装在系统盘，第二个安装在数据盘，避免超过磁盘负荷（系统和数据盘分别为30G和50G）
conda env create -f envs/phmr.yml

If the environment name already exists:

bashconda env update -n gmr -f envs/gmr.yml --prune
conda env update -n phmr -f envs/phmr.yml --prune

1) GMR Environment

bashconda create -n gmr python=3.10 -y
conda activate gmr
cd /root/gpufree-data/every-embodied/07-机器人操作、运动控制/Locomotion/video2robot
pip install -e .
# gmr下还需要安装

pip install loop-rate-limiters
pip install smplx
pip install imageio
pip install mink
pip install rich
pip install imageio[ffmpeg]

2) PromptHMR Environment (4090 Scenario)

bashconda create -n phmr python=3.10 -y
conda activate phmr
cd /root/gpufree-data/every-embodied/07-机器人操作、运动控制/Locomotion/video2robot/third_party/PromptHMR
# bash scripts/install.sh --pt_version=2.4 不推荐使用默认一键安装，bug较多

If automatic installation fails, use the manual approach (see “Manual Configuration Approach” below).

IV. Web UI Launch (Stable Version)

bashconda activate phmr
python -m pip install -U fastapi "uvicorn[standard]" jinja2 python-multipart

pkill -f "video2robot/visualization/robot_viser.py"
cd /root/gpufree-data/every-embodied/07-机器人操作、运动控制/Locomotion/video2robot
export VISER_FIXED_PORT=8789
python -m uvicorn web.app:app --host 0.0.0.0 --port 8000

If conda run -n phmr is not found in GPFree or background tasks, with no write permission, or an abnormal environment name parsing, use the absolute environment path instead:

bashVISER_FIXED_PORT=8789 /root/gpufree-data/conda_envs/phmr/bin/python -m uvicorn web.app:app --host 0.0.0.0 --port 8000

Access: http://localhost:8000

After starting, it is recommended to use the interface to confirm that the Web service is actually available. Do not just check whether uvicorn is running:

bashcurl http://127.0.0.1:8000/health
curl http://127.0.0.1:8000/
curl http://127.0.0.1:8000/api/projects
curl http://127.0.0.1:8000/api/viser/status

Instructions:

Using fixed VISER_FIXED_PORT is to avoid connection rejection due to random port in iframe.
The web interface supports Chinese, Seedance, video uploading, and iron/color switching.
DouBao web interface generates robot video prompts: help me generate images: help me optimize the prompt, create Chinese martial arts videos, with the characters completely within the video frame.
Just upload and run the video; refer to the video tutorial for details.

Some small torch models will be automatically downloaded during execution. If the download fails, try manual download.

mkdir -p /root/.cache/torch/hub/checkpoints && rm -f /root/.cache/torch/hub/v0.10.0.zip.*.partial && wget -O /root/.cache/torch/hub/v0.10.0.zip https://github.com/pytorch/vision/zipball/v0.10.0 && wget -O /root/.cache/torch/hub/checkpoints/deeplabv3_resnet50_coco-cd0a2569.pth https://download.pytorch.org/models/deeplabv3_resnet50_coco-cd0a2569.pth && ls -lh /root/.cache/torch/hub/v0.10.0.zip /root/.cache/torch/hub/checkpoints/deeplabv3_resnet50_coco-cd0a2569.pth

V. Multi-person Trajectory and Visualization

1) Generate multi-track robot actions

bashconda activate gmr
cd /root/gpufree-data/every-embodied/07-机器人操作、运动控制/Locomotion/video2robot
python scripts/convert_to_robot.py --project data/video_001 --all-tracks

2) robot-viser displays all trajectories

bashpython scripts/visualize.py --project data/video_001 --robot-viser --robot-all

VI. MuJoCo Video Export

Single Robot

bashconda activate gmr
cd /root/gpufree-data/every-embodied/07-机器人操作、运动控制/Locomotion/video2robot/third_party/GMR
python scripts/vis_robot_motion.py \
  --robot unitree_g1 \
  --robot_motion_path /root/gpufree-data/every-embodied/07-机器人操作、运动控制/Locomotion/video2robot/data/video_005/robot_motion.pkl \
  --record_video \
  --video_path /root/gpufree-data/every-embodied/07-机器人操作、运动控制/Locomotion/video2robot/data/video_005/mujoco_robot.mp4

Multi-robot (New script)

bashpython scripts/vis_robot_motion_multi.py \
  --robot unitree_g1 \
  --robot_motion_paths \
    /root/gpufree-data/every-embodied/07-机器人操作、运动控制/Locomotion/video2robot/data/video_005/robot_motion_track_1.pkl \
    /root/gpufree-data/every-embodied/07-机器人操作、运动控制/Locomotion/video2robot/data/video_005/robot_motion_track_2.pkl \
  --record_video \
  --max_seconds 10 \
  --camera_azimuth 0 \
  --video_path /root/gpufree-data/every-embodied/07-机器人操作、运动控制/Locomotion/video2robot/data/video_005/mujoco_multi_robot_10s_front.mp4

Instructions:

Fixed the “0 seconds/super short video” issue by strictly controlling the duration based on frames.
Fixed the “only the ground is visible, no robots” issue, where the camera follows the center of multiple robots.

VII. Challenges in Implementation and Prospects of IsaacSim Simulation

Realistic Implementation Challenges

Action mapping accuracy: There are physical limitations in the retargeting of SMPL-X poses extracted from videos to robot joints.
Lack of environmental awareness: Currently, it is mainly about replicating open loop actions, without real-time feedback on actual ground friction, collisions, and obstacles.
Dynamics constraints: Human movements in videos may exceed the motor torque or balance limits of robots. For example, during the martial arts robot performance at the Ma Spring Festival Gala, high-level hovering and force conditions require extremely high servo response capabilities.

Advantages of IsaacSim Simulation

In the future, we will introduce IsaacSim for reinforcement learning training:

Advantages: High physical simulation accuracy, supporting complex collision detection, friction models, and sensor simulation.
Value: For this project, the “action mapping” is mainly completed, while IsaacSim is responsible for “action robustness training”.

VIII. Quick Reference for Common Issues and Debug Records

Q1: `conda` cannot be found (web task timeout)

It has been fixed in the main repository task runner, and automatic detection of CONDA_EXE is supported.

Q2: robot-viser occasional `localhost 拒绝连接`

Use fixed port first: export VISER_FIXED_PORT=8789.

Q3: MuJoCo recording reports that `imageio` backend is missing

Execute: pip install -U "imageio[ffmpeg]".

Q4: Compilation error (lietorch/droid_backends)

For PyTorch 2.4+, .type() should be changed to .scalar_type().

Q5: Unable to download files; download locally

Model file phase (the most critical issue) The errors that often occur when running python scripts/extract_pose.py --project data/video_001 are essentially of the same type:

invalid load key, ‘v’
UnpicklingError
weights_only … Unsupported operand 118
Cannot load file … allow_pickle=False These indicate that the downloaded files cannot be used properly.

Download and verify locally (example)

Download (HF single file)

bash$env:HF_ENDPOINT="https://hf-mirror.com"
$env:HF_HUB_DISABLE_XET="1"

python - << 'PY'
from huggingface_hub import hf_hub_download
p = hf_hub_download(
    repo_id="Datawhale/spring-festival-wushu-robot-replication-model",
    filename="data/pretrain/phmr/checkpoint.ckpt",
    repo_type="model",
    local_dir=r"E:\...\model_tmp",
    force_download=True
)
print("saved:", p)
PY

Local execution may not run the above Python code properly. You need to create a new file.

pythonfrom huggingface_hub import hf_hub_download
p = hf_hub_download(
    repo_id="Datawhale/spring-festival-wushu-robot-replication-model",
    filename="data/pretrain/phmr/checkpoint.ckpt",
    repo_type="model",
    local_dir=r"E:\...\model_tmp",
    force_download=True
)
print("saved:", p)

Save as test.py file python test.py 2) Local verification (must pass)

pythonimport torch
p=r"E:\...\model_tmp\data\pretrain\phmr\checkpoint.ckpt"
with open(p,'rb') as f:
    print("first16:", f.read(16))
obj=torch.load(p, map_location='cpu', weights_only=False)
print("torch.load OK:", type(obj))

Verification criteria

first16 cannot be b’version https://’
torch.load OK shows normal output

Server verification Similar to the above verification methods

Q6: `git lfs clone` is very slow or fails repeatedly

Prioritize placing long-duration download tasks in the tmux backend, and ensure the download directory is correct. If a partial download has already been made, enter the corresponding model repository directory and execute git lfs pull to continue the download. When the domestic network is unstable, try hf-mirror.com, and confirm that the terminal traffic actually passes through a proxy; some proxy software only proxies the browser, not necessarily the remote terminal automatically.

Q7: Web UI stuck at 96%/99%

This is not necessarily a front-end issue. Prioritize checking the backend logs, task processes, and model weights to ensure they are complete, as well as whether the project output files have been generated. When there are missing multi-person videos or model weights, the front-end progress may also remain stationary for a long time.

Q8: `conda run -n phmr/gmr` failed in the GPFree environment

If the environment is actually located in /root/gpufree-data/conda_envs/phmr or /root/gpufree-data/conda_envs/gmr, use an absolute path for execution:

bashconda run -p /root/gpufree-data/conda_envs/phmr ...
conda run -p /root/gpufree-data/conda_envs/gmr ...

9. Warehouse Organization Suggestions (integrate into main warehouse, abandon old warehouse)

It is recommended to maintain video2robot as a regular directory in the main repository, and no longer use submodules (this is the only recommended approach at present).

Reasons:

The old repository hope5hope/video2robot is about to be abandoned. Continuing to use the submodule will increase the risk of failure.
The team wants “tutorials and code in the same repository” to reduce the cost for new members in switching repositories.
Future modifications can be submitted, reviewed, and released directly in the main repository.

Record baseline commit

Main repository: 030f3410dac3cb15a2570376dca6a0f46c2d158c
PromptHMR: 4f8915c5b9603344c56e95fadb9a01a23ba2272d
GMR: 069b4fd48f440e813b2b4d69255c70f53e5f83fb

10. Warehouse Status Description

hope5hope/video2robot is retained as a historical warehouse for reference only and is no longer used as the main development and delivery warehouse.

11. YAML Export and Cross-machine Replication Instructions

Export on the “source machine”:

bashcd /root/gpufree-data/every-embodied/07-机器人操作、运动控制/Locomotion/video2robot
mkdir -p envs
conda env export -n gmr > envs/gmr.yml
conda env export -n phmr > envs/phmr.yml

Import into “Target Machine”:

bashgit clone https://github.com/datawhalechina/every-embodied.git
cd every-embodied/07-机器人操作、运动控制/Locomotion/video2robot
conda env create -f envs/gmr.yml
conda env create -f envs/phmr.yml

Apply 3 more patches (if your clone depends on local changes):

bashgit apply patches/main.patch
git -C third_party/PromptHMR apply ../../patches/prompthmr.patch
git -C third_party/GMR apply ../../patches/gmr.patch

Instructions:

YAML will not automatically include “uncommitted source code changes” and “model weight files”.
Therefore, it is recommended to deliver the YAML + patch + 模型下载脚本 trio together, as this ensures the highest success rate of replication.

Appendix: Reference for Manual Configuration Scheme (phmr environment)

bash# Clone main repo
git clone https://github.com/datawhalechina/every-embodied.git
cd every-embodied/07-机器人操作、运动控制/Locomotion/video2robot


conda create -n phmr python=3.10 -y
conda activate phmr
cd /root/gpufree-data/every-embodied/07-机器人操作、运动控制/Locomotion/video2robot/third_party/PromptHMR
pip install -r requirements.txt

mkdir -p python_libs
git clone https://github.com/Arthur151/chumpy python_libs/chumpy
python -m pip install -e python_libs/chumpy --no-build-isolation

# 在 bashrc 中加入
export PYTHONPATH=$PYTHONPATH:/root/gpufree-data/every-embodied/07-机器人操作、运动控制/Locomotion/video2robot/third_party/PromptHMR
echo 'export PYTHONPATH=$PYTHONPATH:/root/gpufree-data/every-embodied/07-机器人操作、运动控制/Locomotion/video2robot/third_party/PromptHMR' >> ~/.bashrc
source ~/.bashrc


conda install -c conda-forge eigen -y

# export CPATH="$CONDA_PREFIX/include/eigen3:${CPATH:-}" && cd /root/gpufree-data/every-embodied/07-机器人操作、运动控制/Locomotion/video2robot/third_party/PromptHMR/pipeline/droidcalib && python setup.py install
# 编译droidcalib
source /opt/conda/etc/profile.d/conda.sh
conda activate phmr

conda install -c conda-forge eigen -y
export CPATH="$CONDA_PREFIX/include/eigen3:${CPATH:-}"

cd /root/gpufree-data/every-embodied/07-机器人操作、运动控制/Locomotion/video2robot/third_party/PromptHMR/pipeline/droidcalib
python setup.py install

export LD_LIBRARY_PATH="/root/gpufree-data/conda_envs/phmr/lib/python3.10/site-packages/torch/lib:/usr/lib/x86_64-linux-gnu:/usr/local/cuda/lib64:/usr/lib64:/usr/local/lib"
# 上面这个加到bashrc

echo 'export LD_LIBRARY_PATH="/root/gpufree-data/conda_envs/phmr/lib/python3.10/site-packages/torch/lib:/usr/lib/x86_64-linux-gnu:/usr/local/cuda/lib64:/usr/lib64:/usr/local/lib"' >> ~/.bashrc
source ~/.bashrc

# 编译 lietorch
mkdir -p python_libs
cd python_libs
git clone https://github.com/princeton-vl/lietorch.git
cd lietorch
git submodule update --init --recursive
python setup.py install
cd ../..

# 安装 detectron2
git clone https://github.com/facebookresearch/detectron2.git
cd /root/gpufree-data/detectron2
pip install -e . --no-build-isolation

# 安装 SAM2
git clone https://github.com/facebookresearch/segment-anything-2.git
cd segment-anything-2
pip install -e . --no-build-isolation

# 适配代码
sed -i 's/load_video_frames, load_video_frames_from_np/load_video_frames/g' /root/gpufree-data/every-embodied/07-机器人操作、运动控制/Locomotion/video2robot/third_party/PromptHMR/pipeline/detector/sam2_video_predictor.py

补充pip包

python -m pip install -U torch-scatter --no-build-isolation #这个命令运行较久，请耐心等待
python -m pip install -U xformers

Update the download script: https://github.com/hope5hope/video2robot/blob/main/文件更新说明.md

Download relevant model weights (recommended method):

bashgit-lfs install
git lfs clone https://huggingface.co/Datawhale/spring-festival-wushu-robot-replication-model
或者国内镜像
git lfs clone https://hf-mirror.com/Datawhale/spring-festival-wushu-robot-replication-model

The large model download takes a long time. It is recommended to execute it in the tmux backend to avoid interruption due to terminal disconnection. Before downloading, also ensure that the current directory is:

bashcd /root/gpufree-data/every-embodied/07-机器人操作、运动控制/Locomotion/video2robot/third_party/PromptHMR
tmux new-session -d -s phmr_lfs_download 'git lfs clone https://hf-mirror.com/Datawhale/spring-festival-wushu-robot-replication-model'
tmux attach -t phmr_lfs_download

If the repository has already downloaded some files but the LFS files are incomplete, do not delete and re-download directly. Instead, enter the model repository directory to transfer the files later:

bashgit lfs pull

Download relevant model weights (alternative):

bashcd /root/gpufree-data/every-embodied/07-机器人操作、运动控制/Locomotion/video2robot/third_party/PromptHMR
bash scripts/fetch_smplx.sh
bash scripts/fetch_data.sh

Note: fetch_smplx.sh does not only download SMPL-X. The script will also download SMPL and supplementary files. The fact that data/body_models/smplx/SMPLX_NEUTRAL.npz exists does not mean that all model files have been downloaded.

Merge Patch command (for quick synchronization of changes)

If you are syncing based on the original repository, you can use the following command to merge patches:

bashcd /root/gpufree-data/every-embodied/07-机器人操作、运动控制/Locomotion/video2robot

# 1. 确保子模块已初始化
git submodule update --init --recursive

# 2. 应用主仓库补丁
git apply patches/main.patch

# 3. 应用 PromptHMR 补丁
git -C third_party/PromptHMR apply ../../patches/prompthmr.patch

# 4. 应用 GMR 补丁
git -C third_party/GMR apply ../../patches/gmr.patch

Spring Festival Gala Dance Robot Recreation: Complete Practical Record (2026 Version)

1. Project Objectives

II. Core Process

III. Environment Installation (Recommended)

0) Clone the new machine (do this first)

1) GMR Environment

2) PromptHMR Environment (4090 Scenario)

IV. Web UI Launch (Stable Version)

V. Multi-person Trajectory and Visualization

1) Generate multi-track robot actions

2) robot-viser displays all trajectories

VI. MuJoCo Video Export

Single Robot

Multi-robot (New script)

VII. Challenges in Implementation and Prospects of IsaacSim Simulation

Realistic Implementation Challenges

Advantages of IsaacSim Simulation

VIII. Quick Reference for Common Issues and Debug Records

Q1: conda cannot be found (web task timeout)

Q2: robot-viser occasional localhost 拒绝连接

Q3: MuJoCo recording reports that imageio backend is missing

Q4: Compilation error (lietorch/droid_backends)

Q5: Unable to download files; download locally

Q6: git lfs clone is very slow or fails repeatedly

Q7: Web UI stuck at 96%/99%

Q8: conda run -n phmr/gmr failed in the GPFree environment

9. Warehouse Organization Suggestions (integrate into main warehouse, abandon old warehouse)

Record baseline commit

10. Warehouse Status Description

11. YAML Export and Cross-machine Replication Instructions

Appendix: Reference for Manual Configuration Scheme (phmr environment)

Merge Patch command (for quick synchronization of changes)

Q1: `conda` cannot be found (web task timeout)

Q2: robot-viser occasional `localhost 拒绝连接`

Q3: MuJoCo recording reports that `imageio` backend is missing

Q6: `git lfs clone` is very slow or fails repeatedly

Q8: `conda run -n phmr/gmr` failed in the GPFree environment