9) Embodied AI Beginner’s Guide v6

• Open-source content: Every-Embodied: Zero to Hero in Embodied AI
• Learning guide knowledge base: Learning Guide
• Q&A section: Question Records and Summaries
• Check-in information, excellent assignments, and teaching assistant collaboration sheets shall be based on the current Datawhale group announcement.

This version of the path is aimed at the next 20-day team-based learning session. After v5, many new contents have been added to the repository. If we continue to list them separately under “VLA / Vision / CAD / Simulation / Hardware / World model”, beginners may easily lose track of what to do in the first step. v6 is organized again around the four core capabilities of embodied AI systems:

These four directions are not mutually exclusive. A complete robot system will ultimately utilize manipulation, perception, planning, and world model simultaneously. However, during the team learning phase, it is recommended to choose a main direction first and establish a clear workflow.

Course Information

• Target audience: Students, engineers, and researchers interested in embodied AI; basic Python knowledge and command-line experience are recommended.
• Learning period: 20 days, divided into 5 tasks, with 4 days for each task.
• Schedule: The deadline for Task01 is 07 July 03:00, and subsequent tasks will be postponed by 4 days.
• Minimum completion goal: Complete Task01, and complete at least one priority task in Task02 to Task04 in the same direction.
• Recommended completion goals: Complete one priority reproduction and one advanced review, and produce a reusable report for Task05.

How to Choose the Direction

If there is no clear direction yet, you can directly select based on the following judgment:

For those without hardware and high-performance GPUs, prioritize perception mapping or navigation planning; for those with experience with robotic arm, RDK-X5, LeRobot, or MuJoCo, prioritize manipulation control; for those interested in reading papers and analyzing methods, prioritize world model.

General Check-in Rules

For each task, only one assignment in one direction needs to be submitted. Do not run all the links repeatedly.

1. Prioritize reproduction: Submit screenshots, logs, videos, result charts, or a brief reproduction report.
2. Read introduction: Provide a breakdown of the method, explaining task definition, inputs and outputs, core modules, and boundaries.
3. Environment diagnosis: If it fails to run properly, submit system version, GPU/CPU, Python/Conda environment, error locations, and next steps for troubleshooting.
4. Final report: Avoid stacking links; focus on clearly stating the selected approach, evidence of completion, failure points, and next steps.

Task Schedule

Task02: Introduction to Direction

The goal of Task02 is not to build large projects, but to enable each student to complete their first deliverable in one direction.

Task03: Practical Direction Practice

Task03 begins with a more comprehensive reproduction or method reading. Still, only one direction should be selected.

Manipulation Control

Priority selection:

• OpenVLA reproduction
• MuJoCo reproduction of ACT, Pi0, and SmolVLA
• RDK-X5 connecting LeRobot robotic arm for remote manipulation
• EVA-Client real robot deployment and evaluation navigation
• Robots That Know What to Ask reward alignment introduction

The key of delivery is to clearly explain the chain: “observation -> model/policy -> action -> execution or evaluation”. For real hardware tasks, remote manipulation photos, videos, or data collection records can be submitted; for simulation tasks, training/inferencing screenshots and failure cases can be provided. In terms of reward alignment, there is no obligation to reproduce the real robotic arm; instead, focus on explaining the relationship between teaching, feature variance, under-specified detection, interpretive questioning, and reward updates.

Perception Mapping

Priority selection:

• Locate Anything Visual Language Positioning
• LingBot-Map Video Streaming 3D Reconstruction
• Text-to-CAD Engineering Modeling Introduction
• ForgeCAD Visual Reverse Engineering Introduction
• Shape Your Body Value Gradient Robot Design Guide

The focus of the delivery is to explain what the input is, what the model or tool does, and how the output serves the robot. In the CAD direction, the real workflow should be clearly described: natural language or reference images do not directly become CAD. Usually, parametric source code is generated first, and then PNG/GIF/STEP/STL/3MF files are exported by the CAD runtime. In the Shape Your Body direction, the relationship between body parameters, simulation control, value functions, and gradient design should be clearly stated.

Navigation Planning

Preferred choice:

• Habitat Sim Basic Practice
• Habitat Lab Basic Practice
• ETPNav Code Reproduction
• NBS Active Perception Path Planning Guide
• UAV Multimodal Large Model Practice

The focus of delivery is to clarify the relationship between maps, observations, targets, paths, information benefits, and actions. Navigation tasks should not only consider whether the destination is reached, but also record reasons for failure, such as positioning drift, insufficient observation, detours in the path, unclear language targets, or inaccurate estimation of frontier/gain in active perception.

World Model

Priority selection:

• LeWorldModel Analysis, Interpretation, and Reproduction
• RISE Self-Improvement Robot Policy
• RoboDream Composite World Model Data Synthesis
• Gamma-World Multi-Agent World Model
• BWM Action-Dependent World Model
• τ0-WM Unified Video-Action World Model
• GE-Sim 2.0 Closed Loop Video World Simulator

The key focus is to determine which world model it belongs to: predicting latent data, predicting videos, performing imagined RL, enhancing VLA, synthesizing teaching data, generating multi-agent shared worlds, conditioning video rollouts with trajectory conditions, unifying video prediction, action generation, and action evaluation during testing for correction, or packaging the world model as a learned simulator that can be used in closed loop. There is no mandatory reproduction of experiments for the world model direction; clarity in the method boundaries is sufficient for effective work.

Task04: Advanced Review

Task04 is used for comparison or in-depth analysis. Here are recommended topics for each direction.

If you have computing power or hardware capabilities, you can choose these advanced practices:

• SIM1 Soft body simulation and data generation
• UniLab + MotrixSim heterogeneous robot RL
• Genesis World 1.0 full experience and robot simulation pipeline
• AMD ROCm policy replication topic
• AGILE humanoid robot Loco-Manipulation
• HIMLoco quadruped robot motion control
• HumanoidMimicGen full-body planning data generation

Task05: Result Organization and Cross-Share

In the final stage, it is not recommended to create new environments. Just organize the previously completed work into reusable results.

Recommended report structure:

1. The direction I chose: manipulation control / mapping construction / navigation planning / world model.
2. The priority tasks I completed: what was run, what was read, and the results.
3. Key evidence: screenshots, logs, videos, generated maps, trajectories, 3D models, or tables.
4. Method understanding: inputs, outputs, core modules, evaluation metrics, and failure boundaries.
5. Pitfalls encountered: issues with environment, dependencies, memory, data, paths, hardware, or paper comprehension.
6. Advice for the next class: what to do first, what to do later, and which topics can be chosen.

Example of Four Routes

Route A: Manipulation Control

Task01 Run through the minimum demo of the project; Task02 Select MuJoCo or OpenVLA; Task03 Complete reproduction of ACT/Pi0/SmolVLA or OpenVLA; Task04 Read DiT4DiT, EventVLA, WALL-OSS, 3DVLA, PhysBrain 1.0, PRTS, Galaxea G0.5, Dexora, LWD, VisualThink-VLA, Agentic-VLA, Dexbotic-RLinf, DM0.5, EVA-Client or Robots That Know What to Ask; Task05 Output a “VLA/manipulation policy/post-training engineering/high-performance inference/real robot deployment evaluation/reward alignment review”.

Route B: Mapping via Perception

1. Task01: Understand the structure of embodied systems; Task02: Perform SAM/depth estimation or Locate Anything; Task03: Perform LingBot-Map or ForgeCAD; Task04: Perform Locate Anything vs YOLO, Text-to-CAD, 3D model case or Shape Your Body; Task05: Produce a report on “how robots understand scenes and form 3D assets/robot design parameters”.

Route C: Navigation Planning

Task01 Understand robot systems and coordinate transformation; Task02 Run Habitat Sim/Lab; Task03 Perform ETPNav, NBS active perception or multi-modal navigation for drones; Task04 Analyze the path planning, information benefits, language goals, and failure trajectories in Habitat/ETPNav, NBS active perception path planning , drone control or drone planning ; Task05 Produce a review of “navigation system from observation to action”.

Route D: world model

Task01 Understand the difference between the world model and VLA; Task02 Learn VAE/DDPM or LeWM; Task03 Read RISE, RoboDream, Gamma-World, BWM, τ0-WM or GE-Sim 2.0; Task04 Compare RAW-Dream, WoG, WALL-WM, Gamma-World, BWM, τ0-WM, GE-Sim 2.0; Task05 Produce a report explaining “what the world model is predicting and how it serves robots.”

Content Index