8) 2 Embodied AI Beginner’s Guide
- Open-source content: https://github.com/datawhalechina/every-embodied
- Learning guide knowledge base: Learning Guide
- Q&A section: Question records and summaries
- Teaching assistants should apply for editing rights to this document. All information such as check-in records and excellent assignments should be summarized in: 202603 Group 1 and 202603 Group 2
- Course description:
Embodied AI is very trendy. From Mobile ALOHA to Pi0 and OpenVLA, SOTA models are updated almost every week. However, many people want to get involved but get stuck at the first step:
🤯 The materials are too fragmented; unsure where to start learning from?
🤯 The threshold is too high; the mathematical derivations and physics engine are overwhelming?
🤯 Can’t run, and the environment configuration is so complicated that it makes you doubt everything?
Don’t panic! This beginner-friendly tutorial Every-Embodied: Zero to Hero, initiated by Datawhale and refined by the open-source community, is now officially open source!🌟 Our focus is on: easy for beginners, reproducible in engineering, and adaptable to cutting-edge technologies. It takes you from a single line of Python code to gradually installing a “brain” for the robot!🧠
The path for this version remains unchanged with Task01 as originally scheduled, allowing everyone to continue using the existing teaching assistant schedules and check-in links; adjustments will begin from Task02. The overall strategy is: first complete basic general knowledge, then enter different fields based on interests. The latter part is divided into three categories: manipulation world model and world model, navigation model and drone navigation, and applied practice and system deployment. Students who are interested in robotic arm manipulation, grasping, VLA, ACT, Pi0, SmolVLA, OpenVLA, and world model can enter the manipulation world model and world model branch; those who prefer navigation, VLN, scene understanding, drone navigation, and space task planning can enter the navigation model and drone navigation branch; those aiming to develop development boards, teleoperation, simulation tools, and multi-modal demos can enter the applied practice and system deployment branch.
- Target audience: Students, engineers, and researchers interested in embodied AI; those with a foundation in Python and basic mathematical knowledge. Beginners in Python / AI developers: Those who want to systematically transition into embodied AI fields; university students: Those needing course projects or final projects, or those aiming to quickly develop the ability to reproduce research results; hardware/robotics enthusiasts: Those seeking to establish a complete closed loop from physical machine control to Sim2Real deployment.
- Course number: 929
- Number of learning groups: 2
- Number of participants per group: 180
| Role | Name |
|---|---|
| Route pioneer (course designer) | Chen Kewei |
| Navigator (operational teaching assistant) | |
| Sailor (professional teaching assistant) |
Task Schedule:
| Task Information (Choose one link to complete for each task) | Days | Deadline | Review | Excellent Assignment |
|---|---|---|---|---|
| Officially starts on April 13 | Total 20 days | |||
| Task01: Basics of embodied AI and robot control, overview and development of embodied AI, implementation of PID control algorithms, fundamentals of robot kinematics, Preliminary embodied navigation (video tutorial) Corresponding: Popular content: Spring Festival Gala robot replication/every-embodied/blob/main/07-Robot manipulation and motion control/Locomotion/01-Spring Festival Gala dance robot replication.md Video tutorial: http://xhslink.com/o/1lB9dX0Vt2t Course 01-Basics of embodied AI: This chapter covers the overview and development of embodied AI. /datawhalechina/every-embodied/blob/main/01-Basics of embodied AI/01-Basics of embodied AI.md Course 02-Basics of robots and control, hand-eye coordination: This chapter covers “robot basics” (kinematics) and “control” (PID algorithms)./datawhalechina/every-embodied/blob/main/02-Basics of robots and control, hand-eye coordination/01-Robot spatial description and coordinate transformation.md/datawhalechina/every-embodied/blob/main/02-Basics of robots and control, hand-eye coordination/Supplementary code tutorial_Cartpole three control algorithms practical/cartpole modeling and control.mdBasics of embodied navigation (Prioritize completing the assignment):/every-embodied/blob/main/08-Embedded navigation and VLN/02-Simulation environment basics/habitat navigation environment/habitat_lab environment setup and configuration.md/every-embodied/blob/main/08-Embedded navigation and VLN/02-Simulation environment basics/habitat navigation environment/habitat_sim basic practice.md/datawhalechina/every-embodied/blob/main/08-Embedded navigation and VLN/02-Simulation environment basics/habitat navigation environment/habitat_lab basic.md**Video tutorial: [http://xhslink.com/o/4FGHWn2YICu | 4 days | Deadline: April 17, 03:00 | ||
| Task02: Consolidation of general foundations: Introduction to visual perception, simulation environment and training paradigms Task01 has already covered the basics of embodied AI, robot basic control, and preliminary navigation. In this phase, common competencies will be further developed, focusing on understanding visual perception, depth/3D information, reinforcement learning, simulation platforms, and embodied task training paradigms, preparing for subsequent specialized studies. [ Course 04-Computer vision and 3D reconstruction in embodied scenarios: This chapter covers basic topics such as robot vision, depth estimation, SAM, and grasping attention. [ SAM and depth estimation Grabbing attention heatmap Course 05-Depth and reinforcement learning in embodied scenarios: This chapter covers reinforcement learning and basics of embodied decision-making. [ Multi-robot furniture handling reinforcement learning Choose one of the simulation tools: ManiSkill environment simulation configuration GenieSim configuration **GenieSim video tutorial: [http://xhslink.com/o/2tybo79bOsk Recommended completion: Explain the relationship between perception, training, and simulation in embodied tasks; or run a sample using visual/simulation/reinforcement learning. |
4 days | Deadline: April 21, 03:00 | ||
| Task03：Branch Introduction: Select 1 Main Branch Based on Interest and Run the First Directional Task After completing Task01 and Task02, start dividing tasks according to interests. It is recommended that each student choose 1 main branch first; pursuing multiple directions simultaneously is not advised. Branch A: Manipulation Model and World Model This is suitable for students who enjoy robotic arm manipulation, grasping, imitation learning, VLA, ACT, Pi0, SmolVLA, OpenVLA, and world models. The manipulation model focuses on “how robots generate actions based on vision and language,” while the world model focuses on “how systems understand, predict, and plan environmental changes.” Both serve the ability to manipulate robots. SmolVLA-LIBERO OpenVLA reproduction MuJoCo reproduction of ACT, Pi0, SmolVLA **MuJoCo grasping experiment video tutorial:https://www.bilibili.com/video/BV1WhxeznE61/ ACT, Pi0 data collection and algorithm practical video tutorials:**http://xhslink.com/o/8lsptSLOlex LIBERO benchmark LeWorldModel analysis, interpretation, and experimental reproduction Branch B: Navigation Model and UAV navigation This is suitable for students who enjoy navigation, VLN, scene understanding, spatial positioning, path planning, UAV navigation, and multi-modal tasks with aerial perspectives. UAV-related content falls under navigation models, as its core is also “understanding targets, planning paths, and executing movements in the environment.” Basic introduction to embodied navigation algorithms Habitat Lab setup and configuration Basic practice of Habitat Sim **Habitat navigation video tutorial:**http://xhslink.com/o/4FGHWn2YICu UAV multi-modal large model practice Corresponding simulator download links (Windows): https://huggingface.co/datasets/Datawhale/house_win Branch C: Application practice and system deployment This is suitable for those who want to develop simulation tools, development boards, real robotic arms, teleoperation, and system integration. RDK-X5 beginner tutorial Connecting RDK-X5 to the LeRobot robotic arm for teleoperation Recommended actions: Choose a main branch, explain the reason for selection, and run the first demo or environment in that direction. |
4 days | Deadline: April 25, 03:00 | ||
| Task04: Deep Dive into Branches: Add new content along the direction of Task03 to produce a reproducible result Task03 has begun, so the learning requirements for Task03 will not be adjusted in this phase. Task04 does not repeat the introductory link of Task03; instead, it provides further in-depth content or additional supplements. Learners should continue to progress along the main branch chosen in Task03, upgrading their output from “running through the first demo or environment” to “reproducible results”: they can choose one of the following activities—training, evaluation, parameter comparison, deployment process, data generation, failure analysis, or in-depth study of papers/code—and their submission should include the process, results, and conclusions. Manipulation model and world model direction: Suitable for students who have already interacted with SmolVLA, OpenVLA, MuJoCo, LIBERO, or LeWorldModel in Task03 to continue their research. For Task04, it is recommended to focus on more systematic model understanding, data generation, or reproduction on new platforms, to avoid submitting the same demo repeatedly. RT Series Paper Analysis and Code Analysis SIM1 Soft Body Simulation and Data Generation Isaac GR00T Introduction and Tuning Navigation model and drone navigation direction: Suitable for students who have already completed the Habitat basic environment, navigation algorithm basics, or drone demo in Task03 to continue deeper study. For Task04, it is recommended to focus on more comprehensive navigation task reproduction, trajectory analysis, organization of success/failure cases, or explaining how VLN/navigation models convert language goals into path decisions. Habitat Lab Basic Practices Comprehensive Introduction to Embodied Navigation Practice Reproduction of ETPNav code **ETPNav Frontier VLN Navigation Algorithm Reproduction Video Tutorial:**http://xhslink.com/o/8t08X6dROt5 Application practice and system deployment direction: This section is suitable for those who have already dealt with RDK-X5, LeRobot, simulation tools, or real system links in Task03 to continue their in-depth study. For Task04, it is recommended to choose topics related to data synthesis, simulation data generation, or system deployment. Focus on providing reproducible steps, verification results, and troubleshooting notes. InternDataEngine Small Space Feature Experience Tutorial SIM1 Soft Body Simulation and Data Generation Isaac GR00T Introduction and Tuning Recommended Submission: Submit “in-depth results of the same branch”. The content should include the continuation of Task03, selected supplementary materials or detailed tutorials, steps for reproduction/training/deployment/data generation, screenshots or metrics of key results, issues encountered and solutions, as well as judgments on further in-depth exploration in this area. |
4 days | Deadline: 03:00 on April 29 | ||
| Task05: Result organization and cross-sharing In the final stage, it is not recommended to start new projects. Instead, organize the previously completed general knowledge learning and branch practices into reusable results. You can submit reproduction reports, demonstration videos, animated images, environment setup records, direction summaries, paper/code interpretations, and summaries of encountered issues. Manipulation model and world model direction: Can organize learning reports for ACT, Pi0, SmolVLA, OpenVLA, RT series, LIBERO, or LeWorldModel. Navigation models and drone navigation directions: Can organize reports on Habitat, VLN, ETPNav, drone navigation, or drone multimodal practice reproduction. Application practice and system deployment direction: Organize environment configurations and common pitfalls encountered in ManiSkill, GenieSim, RDK-X5, LeRobot teleoperation, real system deployment, or multi-modal demos. Suggested submission: Submit a final report describing the branch you chose, tasks completed, issues encountered, solutions adopted, and areas for further exploration. |
4 days | Deadline: May 3, 03:00 |