8) 2 Embodied AI: Beginner’s Guide
- Open-source content: https://github.com/datawhalechina/every-embodied
- Learning guide knowledge base: Learning Guide
- Q&A section: Question records and summaries
- Teaching assistants should apply for editing rights to this document. All information such as check-in records and excellent assignments should be summarized in: Group 1, 202603 and Group 2, 202603
- Course description:
Embodied AI is very trendy. From Mobile ALOHA to Pi0 and OpenVLA, SOTA models are updated almost every week. However, many people want to get involved but get stuck at the first step:
🤯 The materials are too fragmented; not sure where to start learning from?
🤯 The threshold is too high; the mathematical derivations and physics engine are exhausting?
🤯 Can’t run, and the environment configuration is so confusing?
Don’t panic! This beginner-friendly tutorial, Every-Embodied: Zero to Hero, initiated by Datawhale and refined by the open-source community, has been officially opened sourced!🌟 Our focus is on: easy for beginners, reproducible in engineering, and adaptable to cutting-edge technologies. It takes you from a single line of Python code, step by step, to installing the “brain” for the robot!🧠
This version adds priority reproduction suggestions and advanced reproduction suggestions on top of V3’s “start with general knowledge, then branch out” approach. The default rule is: for each Task, prioritize completing one reproduction that is the most stable, classic, and easy to achieve results; other platforms, models, and environments can be chosen as advanced options. This helps beginners avoid the dilemma of not knowing where to start when dealing with multiple environments such as ManiSkill, GenieSim, Isaac Sim, Habitat, OpenVLA, SmolVLA, GR00T, and SIM1 simultaneously.
The latter part is divided into three categories: manipulation model and world model, navigation model and drone navigation, and applied practices and system deployment. Those who are interested in robotic arm manipulation, grasping, VLA, ACT, Pi0, SmolVLA, OpenVLA, and world models can enter the manipulation model and world model branch; those who prefer navigation, VLN, scene understanding, drone navigation, and spatial task planning can enter the navigation model and drone navigation branch; those aiming to develop development boards, teleoperation, simulation tools, data synthesis, and multi-modal demos can enter the applied practices and system deployment branch.
- Target audience: Students, engineers, and researchers interested in embodied AI; those with a basic understanding of Python and mathematical knowledge. Beginners in Python / AI developers: Looking to systematically transition into embodied AI fields; university students: Need to complete course projects or final projects, or want to quickly develop the ability to reproduce research results; hardware/robotics enthusiasts: Want to establish a complete closed loop from physical machine control to Sim2Real deployment.
- Course number: 929
- Number of learning groups: 2
- Number of participants per group: 180
| Role | Name |
|---|---|
| Route opener (course designer) | Chen Kewei |
| Navigator (operational teaching assistant) | |
| Sailor (professional teaching assistant) |
Task Schedule:
| Task information (choose one link to check in for each task) | Number of days | Deadline | Review | Excellent assignment |
|---|---|---|---|---|
| Officially starts on April 13 | 20 days in total | |||
| Task01: Basics of embodied AI and robot control, overview and development of embodied AI, implementation of PID control algorithms, fundamentals of robot kinematics, Preliminary hand-eye navigation (video tutorial) Corresponding: Popular content: Recreation of Spring Festival Gala robots Video tutorial: http://xhslink.com/o/1lB9dX0Vt2t Course 01 – Overview of embodied AI: This chapter covers the overview and development of embodied AI. Overview of embodied AI Course 02 – Basics of robots and control, hand-eye coordination: This chapter covers “robotics basics” (kinematics) and “control” (PID algorithms). Robot spatial description and coordinate transformation Cartpole modeling and control Basics of hand-eye navigation ( Prioritize completing the assignment ): Habitat Sim environment setup and dataset introduction Habitat Lab environment setup and configuration Video tutorial: http://xhslink.com/o/4FGHWn2YICu |
4 days | Deadline: April 17, 03:00 | ||
| Task02: Foundation of General Knowledge: Introduction to Visual Perception, Simulation Environment, and Training Paradigms Task01 has already covered an overview of embodied AI and basic control and navigation for robots. In this phase, we continue to build common capabilities, focusing on understanding visual perception, depth/3D information, reinforcement learning, simulation platforms, and training paradigms for embodied tasks, preparing for subsequent specialized studies. Recommended priority for reproduction:Beginners only need to complete a lightweight example during this phase; there is no requirement to install multiple simulation environments simultaneously. It is recommended to prioritize either visual perception or basic simulation. Course 04: Computer Vision and 3D Reconstruction in Embodied Scenarios:This section covers basic topics such as robot vision, depth estimation, SAM, and grasping attention. SAM and depth estimation Grasing attention heatmap Course 05: Depth and Reinforcement Learning in Embodied Scenarios:This section covers reinforcement learning and basics of embodied decision-making. Reinforcement learning for multi-robot furniture handling Simulation tool selection:Choose the one you find easier to install first. ManiSkill environment simulation configuration GenieSim configuration GenieSim video tutorials:http://xhslink.com/o/2tybo79bOsk Recommended actions:Explain the relationship among perception, training, and simulation in embodied tasks; or run a sample using visual/Simulation/Reinforcement learning. |
4 days | Deadline: April 21st, 03:00 | ||
| Task03: Introduction to Branching – Select 1 Main Branch and Prioritize Completing a Classic Dataset After completing Task01 and Task02, start dividing tasks based on interests. The focus of Task03 is “completing one classic task first”; it is not recommended to try multiple environments simultaneously. Each student only needs to select 1 main branch and prioritize completing 1 recommended dataset within that branch. Branch A: Manipulation Model and World Model Suitable for students who enjoy robotic arm manipulation, grasping, imitation learning, VLA, ACT, Pi0, SmolVLA, OpenVLA, and world models. Recommended priority dataset reproduction: choose one of the two options. 1. OpenVLA Dataset Reproduction (Priority): Classic VLA introduction, suitable for those who want to understand “how visual language outputs actions”. 2. MuJoCo Dataset Reproduction of ACT, Pi0, SmolVLA (Priority): Classic manipulation simulation introduction, suitable for those who want to practice grasping, ACT, Pi0, SmolVLA. MuJoCo grasping experiment video tutorial:https://www.bilibili.com/video/BV1WhxeznE61/ ACT and Pi0 data collection and algorithm practice video tutorials:http://xhslink.com/o/8lsptSLOlex Advanced reading recommendations:Review these after completing the priority reproduction, as they are not mandatory for Task03. SmolVLA-LIBERO LIBERO benchmark LeWorldModel analysis and experimental reproduction Branch B: Navigation Model and Drone Navigation Suitable for students who enjoy navigation, VLN, scene understanding, spatial positioning, path planning, drone navigation, and multi-modal tasks with aerial perspectives. Recommended priority dataset reproduction:Prioritize completing the basic Habitat links; do not focus on both VLN cutting-edge and drone tasks from the start. Students without properly configured environments should first set up the environment; those with a proper setup can proceed directly with basic practices. Habitat Sim Environment Setup and Dataset Introduction Habitat Lab Environment Setup and Configuration Habitat Sim Basic Practices (Priority) Habitat Lab Basic Practices (Priority) Habitat Navigation Video Tutorial:http://xhslink.com/o/4FGHWn2YICu Advanced reading recommendations:Choose after completing the basic Habitat setup. Basic Introduction to Embodied Navigation Algorithms Drone Multi-modal Large Model Practice Drone Control Tutorial Drone Planning Tutorial Corresponding simulator download links (Windows):https://huggingface.co/datasets/Datawhale/house_win Branch C: Application Practice and System Deployment Suitable for those who want to work on development boards, real robotic arms, teleoperation, simulation tools, data synthesis, and system integration. Recommended priority dataset reproduction:Students without hardware should prioritize ManiSkill or GenieSim; those with RDK-X5 / LeRobot devices should focus on hardware links; those interested in data synthesis and scenario tools can choose InternDataEngine. ManiSkill Environment Simulation Configuration GenieSim Configuration RDK-X5 Super Beginner Tutorial Connecting RDK-X5 to LeRobot robotic arm for teleoperation InternDataEngine Small Space Function Experience Tutorial Advanced reading recommendations:Choose after completing a basic environment or hardware link. Isaac Sim Configuration Instructions SIM1 Soft Body Simulation and Data Generation Recommended checkpoint:Select a main branch, explain the reason for selection, and submit only the result of one priority dataset task; if you have already completed the priority task, you can add an advanced reading or reproduction plan. |
4 days | Deadline: April 25, 03:00 | ||
| Task04: Deep Dive into Branches: Complete an advanced reproduction or comparative analysis along the direction of Task03 Task04 does not require changing directions, nor does it repeat the introductory tasks of Task03. Learners should continue advancing along the main branch selected in Task03, upgrading their output from “running a demo” to “reproducible results”. It is recommended to choose one of the following advanced topics. Branch A: Advanced Manipulation Model and World Model Suitable for those who have already completed the reproduction of OpenVLA or MuJoCo operations in Task03. Task04 does not repeat the introductory steps of OpenVLA/MuJoCo, but focuses on papers, benchmarks, world models, or more comprehensive data/training understanding. Recommended advanced topics: Choose one of the following. 1. Analysis and code review of RT series papers (priority for reproduction): Suitable for clarifying the structure, data format, and inference process of VLA manipulation models. 2. Analysis and experiment reproduction of LeWorldModel (priority for reproduction): Suitable for entering the world model area and understanding the relationship between environmental prediction, planning, and robot manipulation. Additional advanced suggestions:For those interested in benchmarks, SmolVLA, GR00T, soft body simulation, or world model competitions, choose further. LIBERO benchmark SmolVLA-LIBERO Introduction and tuning of Isaac GR00T SIM1 soft body simulation and data generation 1X world model challenge Branch B: Advanced Navigation Model and UAV navigation Suitable for those who have already completed the basic Habitat links in Task03. Task04 does not repeat the Habitat environment configuration and basic practices, but focuses on integrated navigation, cutting-edge VLN, or UAV topics. Recommended advanced topics:Prioritize the comprehensive introduction to embodied navigation practice, including task configuration, trajectory results, failure cases, and improvement suggestions. Comprehensive introduction to embodied navigation practice (priority for reproduction) Additional advanced suggestions:For those more interested in cutting-edge VLN or aerial robots, choose further. ETPNav code reproduction Video tutorial on reproducing cutting-edge VLN navigation algorithms for ETPNav: http://xhslink.com/o/8t08X6dROt5 Practice of multi-modal large models for UAVs Corresponding simulator download links (Windows): https://huggingface.co/datasets/Datawhale/house_win UAV control tutorial UAV planning tutorial Branch C: Advanced application practice and system deployment Suitable for those who have already completed an simulation environment, development board, teleoperation, or data synthesis link in Task03. The focus of Task04 is not to switch to a new environment, but to clarify the system chain. Recommended advanced topics: Choose one of the following. 1. SIM1 soft body simulation and data generation: Suitable for data generation, soft body simulation, and complex manipulation scenarios. 2. Introduction and tuning of Isaac GR00T: Suitable for model tuning, basic robot models, and Isaac ecosystem practices. Additional advanced suggestions:For those requiring cloud deployment, Isaac Sim, or data toolchains, choose further. Configuration instructions for Isaac Sim Complete tutorial on deploying Isaac Lab + GR00T on Alibaba Cloud Tutorial on experiencing the small-space functionality of InternDataEngine Suggested submission:Submit “in-depth results of the same branch”. The content should include the continuity of Task03, the selected advanced reproduction, steps for reproduction/training/deployment/data generation, key result screenshots or metrics, issues encountered and solutions, and directions for further exploration. |
4 days | Deadline: April 29, 03:00 | ||
| Task05: Organization and Cross-Share of Achievements In the final stage, it is not recommended to create a new environment. Instead, organize the previously completed general knowledge learning and branch practices into reusable results. You can submit reproduction reports, demonstration videos, animated images, environment setup records, direction summaries, paper/code interpretations, and summary of common pitfalls. Preferred form of results:It is recommended to submit a combination of “one priority reproduction + one advanced understanding,” rather than listing many failed links. Manipulation model and world model direction:You can organize learning reports on OpenVLA, MuJoCo ACT/Pi0/SmolVLA, RT series, GR00T, LIBERO, SIM1, or LeWorldModel. Navigation model and drone navigation direction:You can organize reports on Habitat, VLN, ETPNav, drone navigation, drone control/planning, or multi-modal drone practice reproduction. Application practice and system deployment direction:You can organize environment configurations and summary of common pitfalls for ManiSkill, GenieSim, Isaac Sim, InternDataEngine, RDK-X5, LeRobot teleoperation, real system deployment, or multi-modal demos. Suggested submission:Submit a final result explaining the branch you selected, priority reproduction tasks, advanced reproduction tasks, problems encountered, solutions, and directions for further exploration. |
4 days | Deadline: May 03, 03:00 |