logo Mr.Robota 具身智能学习手册
/
中文
Chapter 16: Damo Institute & Datawhale Online Team Learning

LightNav-0 Multi-scene Navigation Learning

Use the official pre-trained LightNav-0 to run visual language navigation on a GPU［graphics processor］. No navigation model is trained, and no development board is connected. The accompanying Notebook［interactive tutorial］, Book 8, provides default paths, model services, three scenarios, real predictions, videos, and trajectory comparisons. This is Day 8 of the 8-day learning plan , focusing on practices related to images, language commands, trajectory, and walking control.

Prepare the environment

The model and simulator use two environments, and the weights and environment are placed outside the repository. Here, Ubuntu is used as an example:

bashgit clone https://github.com/lightorigins/LightNav-0.git ~/projects/LightNav-0
uv venv --python 3.11 /data/Data14TB/envs/lightnav-0
uv pip install --python /data/Data14TB/envs/lightnav-0/bin/python -e "$HOME/projects/LightNav-0[vllm,video]"
/data/Data14TB/envs/lightnav-0/bin/hf download LightOriginsHQ/LightNav-0 \
  --local-dir /data/Data14TB/checkpoints/LightNav-0

uv venv --python 3.11 /data/Data14TB/envs/lightnav-mujoco
uv pip install --python /data/Data14TB/envs/lightnav-mujoco/bin/python \
  -e "$HOME/projects/LightNav-0/mujoco_demo[microduck]" \
  jupyterlab nbclient nbformat nbconvert ipykernel matplotlib imageio imageio-ffmpeg websocket-client

Continue to use the official MicroDuck robot resources and walking policy:

bashgit clone https://github.com/pollen-robotics/microduck_rl.git ~/workspaces/microduck_rl
mkdir -p /data/Data14TB/checkpoints/microduck-navigation
curl -fL https://huggingface.co/pollen-robotics/microduck-policies/resolve/main/alpha_walking.onnx \
  -o /data/Data14TB/checkpoints/microduck-navigation/alpha_walking.onnx

Existing environments, code, and weights are reused directly. Model inference uses the fixed vllm==0.19.1 from the upstream, and the simulator requires MuJoCo and scene resources. Use the appropriate accelerated computing environment as per the official instructions, without specifying the device model.

Open and Execute

bash/data/Data14TB/envs/lightnav-mujoco/bin/python -m ipykernel install --user \
  --name microduck-lightnav --display-name 'MicroDuck LightNav'
cd '<专题路径>'
/data/Data14TB/envs/lightnav-mujoco/bin/jupyter lab --no-browser --ip=127.0.0.1 --port=8890

Open 03-Notebook/08_LightNav0_视觉语言导航_GPU_多场景.ipynb, select the above kernel, and run it from top to bottom. The default parameters are already filled in. For other machines, set LIGHTNAV_ROOT, LIGHTNAV_MODEL_ENV, LIGHTNAV_MODEL_PATH, LIGHTNAV_ROBOT_MODEL, LIGHTNAV_WALKING_POLICY, and LIGHTNAV_OUTPUT. The default record is stored in /data/Data14TB/lightnav-recordings/teaching/notebook; a small-sized video copy is written to 03-Notebook/outputs/lightnav_gpu for web playback. This directory is not committed to the repository. When started, use the special catalog as the root directory of the web page, and the relative links between tutorials and supporting code will work properly.

Use for complete batch execution:

bashcd '<专题路径>/03-Notebook'
/data/Data14TB/envs/lightnav-mujoco/bin/jupyter nbconvert --execute --to notebook --inplace \
  --ExecutePreprocessor.timeout=1200 --ExecutePreprocessor.kernel_name=microduck-lightnav \
  '08_LightNav0_视觉语言导航_GPU_多场景.ipynb'
/data/Data14TB/envs/lightnav-mujoco/bin/jupyter trust '08_LightNav0_视觉语言导航_GPU_多场景.ipynb'

Scenario and Control

Scenario Learning Content
Official Residence Official ProcTHOR val_2 multi-room scenario and model interface
Self-built Living Room Rearrange the room, armchair, sofa, and plants to observe the target approaching
Self-built Corridor Narrow space and side furniture; observe local paths and turns

The navigation model reads the first-person view image and instructions, then outputs a 10×3 plane trajectory. MPC［Model Predictive Control］ uses the model trajectory to determine velocity, and the official walking policy outputs 14-dimensional joint actions through the CPU［Central Processing Unit］ interface of ONNX Runtime［Model Inference Runtime］. Gravity, contact, and position actuators remain using the official implementation. Simulation is paused during model inference; videos are encoded based on simulation time, and reports record the actual inference delay separately. The target coordinates are used only for distance measurement, with no input into the model or used for planning control.

Each run overwrites the *_latest.mp4 for each scenario, retaining the corresponding predicted text, trajectory, stop flag, and experiment report. No large videos are embedded in the interactive tutorial; small-sized images and text outputs are retained for easy direct reading.

Files and Sources

  • lightnav_lesson.py: New scenario construction, model protocol, physical closed loop, and video recording.
  • build_notebook.py: Generating tutorials in a structured format; re-generation will clear old execution outputs, followed by a full execution.
  • Official code: LightNav-0.
  • Scenario and robot interface: Official simulator.
  • Navigation weight: LightOriginsHQ/LightNav-0.
  • Robot: pollen-robotics/microduck_rl.

The official scene grid adopts CC BY 4.0; the self-built scene modifies the wall, floor, and furniture layouts, and the modification records are saved along with the output. The robot grid follows the upstream CC BY-NC-SA terms, and the code and policy follow their respective upstream licenses.