8) 2 Embodied AI: Beginner’s Guide
- Open-source content: https://github.com/datawhalechina/every-embodied
- Learning guide knowledge base: Learning Guide
- Q&A section: Question records and summaries
- Teaching assistants should apply for editing rights to this document. All contents such as check-in information and excellent assignments should be summarized in: 1st group, 202603 and 2nd group, 202603
- Course description:
Embodied AI is very trendy. From Mobile ALOHA to Pi0 and OpenVLA, SOTA models are updated almost every week. However, many people want to get involved but get stuck at the first step:
🤯 The materials are too fragmented; not sure where to start learning from?
🤯 The threshold is too high; the mathematical derivations and physics engine are exhausting?
🤯 Can’t run, and the environment configuration is so complex that it makes you doubt everything?
Don’t panic! The beginner-friendly tutorial Every-Embodied: Zero to Hero, initiated by Datawhale and refined by the open-source community, has been officially opened sourced!🌟 Our focus is on: easy for beginners, reproducible in engineering, and adaptable to cutting-edge developments. It takes you from a single line of Python code, step by step, to equipping the robot with a “brain”!🧠
This learning path is adjusted to “begin with general knowledge, then branch out”: In the first half, study general knowledge topics such as an overview of embodied AI, robotics basics, control, visual perception, simulation, and reinforcement learning; in the second half, choose different directions based on interests. The branches no longer combine grasping, navigation, and cutting-edge reproduction, but are divided into three categories: manipulation models and world model, navigation models and drone navigation, and applied practice and system deployment. Those who enjoy robotic arm manipulation, grasping, VLA, ACT, Pi0, SmolVLA, OpenVLA, and world model can enter the manipulation models and world model branch; those who are interested in navigation, VLN, scene understanding, drone navigation, and spatial task planning can enter the navigation models and drone navigation branch; those who want to develop development boards, teleoperation, simulation tools, and multi-modal demos can enter the applied practice and system deployment branch.
- Target audience: Students, engineers, and researchers interested in embodied AI; those with a foundation in Python and basic mathematical knowledge. Beginners in Python / AI developers: Those who want to systematically transition into embodied AI fields; university students: Those needing course projects or final projects, or those aiming to quickly develop the ability to reproduce research results; hardware/robotics enthusiasts: Those seeking to establish a complete closed loop from physical machine control to Sim2Real deployment.
- Course ID: 929
- Number of learning groups: 2
- Number of participants per group: 180
| Role | Name |
|---|---|
| Route opener (course designer) | Chen Kewei |
| Navigator (operational teaching assistant) | |
| Sailor (professional teaching assistant) |
Task Schedule:
| Task Information (Choose one link to check in for each task) | Days | Deadline | Review | Excellent Assignment |
|---|---|---|---|---|
| Officially starts on April 13 | Total 20 days | |||
| Task01: General Foundation I: Overview of Embodied AI and Robot Basics All students should study together first to establish a closed loop understanding of “perception–decision–control”, and fill in the gaps in robot spatial description, coordinate transformation, kinematics, and basic control concepts. Course 01 – Overview of Embodied AI: This chapter covers “what embodied AI is, its development history, typical tasks, and technology landscape”. Overview of Embodied AI Course 02 – Robot Basics and Control, Hand-eye Coordination: This chapter covers robot basics, coordinate transformation, PID control, and simulation practice. Robot Spatial Description and Coordinate Transformation PID Control Basics Comprehensive Practice and Simulation Suggested Check-in: Prepare a basic notes on embodied AI to explain how you understand the relationship between perception, decision-making, control, and robot body; or run a basic control/simulation example and submit screenshots or environment records. |
4 days | Deadline: April 17, 03:00 | ||
| Task02: General Foundation II: Visual Perception, Simulation Environment, and Training Paradigms All students should continue studying together. There is no requirement to delve deeply into a specific direction from the start; instead, focus on understanding common visual inputs, depth estimation, 3D information, reinforcement learning, and simulation platforms in embodied tasks. Course 04 – Computer Vision and 3D Reconstruction in Embodied Scenarios: This chapter covers basic topics such as robot vision, depth estimation, SAM, and grasping attention. SAM and Depth Estimation Grabbing Attention Heatmap Course 05 – Depth and Reinforcement Learning in Embodied Scenarios: This chapter covers reinforcement learning and introductory embodied decision-making. Reinforcement Learning for Multi-Robot Furniture Handling Choose one of the simulation tools: ManiSkill Environment Simulation Configuration GenieSim Configuration Suggested Check-in: Explain the relationship between perception, training, and simulation in embodied tasks; or run a visual/simulation/reinforcement learning example. |
4 days | Deadline: April 21, 03:00 | ||
| Task03：Branch Introduction: Select 1 Main Branch Based on Interest and Run the First Directional Task After completing the first two general knowledge tasks, start dividing into different branches based on interests. It is recommended that each student choose 1 main branch first; pursuing multiple directions simultaneously is not advised. Branch A: Manipulation Model and World Model Suitable for students who enjoy robotic arm manipulation, grasping, imitation learning, VLA, ACT, Pi0, SmolVLA, OpenVLA, and world model. The manipulation model focuses on “how robots generate actions based on vision and language,” while the world model focuses on “how systems understand, predict, and plan environmental changes.” Both serve the ability to manipulate robots. SmolVLA-LIBERO OpenVLA reproduction MuJoCo for ACT, Pi0, SmolVLA LIBERO benchmark LeWorldModel analysis and experiment reproduction Branch B: Navigation Model and UAV Navigation Suitable for students who enjoy navigation, VLN, scene understanding, spatial positioning, path planning, UAV navigation, and multi-modal tasks in aerial perspectives. UAV-related content falls under the navigation model category, as its core is also “understanding targets, planning paths, and performing movements in the environment.” Basic introduction to embodied navigation algorithms Habitat Lab environment setup and configuration Basic practice with Habitat Sim Practice with multi-modal large models for UAVs Download links for corresponding simulators (Windows): https://huggingface.co/datasets/Datawhale/house_win Branch C: Application Practice and System Deployment Suitable for those who want to develop simulation tools, development boards, real robotic arms, teleoperation, and system integration. RDK-X5 beginner tutorial Connecting LeRobot robotic arm with RDK-X5 for teleoperation Recommended actions: Choose a main branch, explain the reason for selection, and run the first demo or environment in that direction. |
4 days | Deadline: April 25, 03:00 | ||
| Task04：Deep Dive into Sub-tasks: Continue Deepening Along the Direction Selected in Task03 It is not recommended to frequently switch directions during this stage. The focus should be on completing a more complete reproduction, training, deployment, comparative analysis, or troubleshooting summary in the same direction. Manipulation model and world model direction: You can continue learning about the RT series, ACT/Pi0/SmolVLA training and deployment, LIBERO benchmark, or LeWorldModel world model topics. Interpretation of RT series papers and code analysis Analysis and interpretation of LeWorldModel and experimental reproduction Navigation model and drone navigation direction: You can continue completing basic practices in Habitat, comprehensive navigation practices, cutting-edge reproduction of ETPNav, or multi-modal practices for drones. Basic practices in Habitat Lab Comprehensive entry-level practices in embodied navigation Code reproduction of ETPNav Multi-modal large-scale model practices for drones Application practices and system deployment direction: You can continue deepening your knowledge in ManiSkill, GenieSim, RDK-X5, LeRobot manipulation, or real system integration. Simulation configuration for ManiSkill environment Configuration of GenieSim Connecting RDK-X5 with a robotic arm for manipulation Recommendation: Complete a more complete reproduction, training, deployment, comparative analysis, or troubleshooting summary in the same direction. |
4 days | Deadline: April 29, 03:00 | ||
| Task05：Result Organization and Cross-Share In the final stage, it is not recommended to start new projects. Instead, organize the general knowledge learning and sub-task practices completed earlier into reusable results. You can submit reproduction reports, demonstration videos, animated images, environment setup records, direction summaries, paper/code interpretations, and troubleshooting summaries. Manipulation model and world model direction: You can organize learning reports on ACT, Pi0, SmolVLA, OpenVLA, RT series, LIBERO, or LeWorldModel. Navigation model and drone navigation direction: You can organize reproduction reports on Habitat, VLN, ETPNav, drone navigation, or multi-modal drone practices. Application practices and system deployment direction: You can organize environment configurations and troubleshooting summaries for ManiSkill, GenieSim, RDK-X5, LeRobot manipulation, real system deployment, or multi-modal demos. Recommendation: Submit a final result explaining the branch you chose, the tasks completed, the problems encountered, the solutions, and the directions for further exploration. |
4 days | Deadline: May 03, 03:00 |