8) 2 Embodied AI Beginner’s Guide
- Open-source content: https://github.com/datawhalechina/every-embodied
- Learning guide knowledge base: Learning Guide
- Q&A section: Issue records and summaries
- The teaching assistant should apply for editing rights to this document. All contents such as check-in information and excellent assignments should be summarized in: 1st group, 202603 and 2nd group, 202603
- Course description:
Embodied AI is very trendy. From Mobile ALOHA to Pi0 and OpenVLA, SOTA models are updated almost every week. However, many people want to get involved but get stuck at the first step:
🤯 The materials are too fragmented; not sure where to start learning from?
🤯 The threshold is too high; the mathematical derivations and physics engine are exhausting?
🤯 Can’t run, and the environment configuration is so complicated that it makes you doubt everything?
Don’t panic! This beginner-friendly tutorial, Every-Embodied: Zero to Hero, initiated by Datawhale and refined by the open-source community, is now officially open source!🌟 Our focus is on: easy for beginners, reproducible in engineering, and adaptable to cutting-edge technologies. It takes you from a single line of Python code to gradually installing a “brain” for the robot!🧠
Reject the “environment setup from basics to abandonment” approach! We provide ready-to-use environment scripts. Even with only basic programming skills, just three command steps are enough to run the first MuJoCo robotic arm grasping demo locally. We will help you get OpenVLA, SmolVLA, ETPNav working, and even replicate the martial arts robot featured at the Spring Festival Gala! Additionally, 🎬 there are beginner-friendly video tutorials! We have updated numerous practical videos on REDnote and B 站 simultaneously! From Mujoco simulation, ACT grasping water cup algorithms to GenieSim one-click startup, we will guide you step by step to avoid common pitfalls.
- Target audience: Students, engineers, and researchers interested in embodied AI; those with a foundation in Python and basic mathematical knowledge. Beginners in Python / AI developers: Those who want to systematically transition into embodied AI fields; university students: Those needing course projects or final projects, or those aiming to quickly develop the ability to reproduce research results; hardware/robotics enthusiasts: Those seeking to establish a complete closed loop from physical machine control to Sim2Real deployment.
- Course ID: 929
- Number of learning groups: 2
- Number of participants per group: 180
| Role | Name |
|---|---|
| Route Developer (Course Designer) | Chen Kewei |
| Navigator (Operational Tutor) | |
| Sailor (Professional Tutor) |
Task arrangement:
| Task Information (Choose one link to complete for each task) | Days | Deadline | Review | Excellent Assignment |
|---|---|---|---|---|
| Officially starts on April 13 | Total 20 days | |||
| Task01: Foundations of Embodied AI and Robot Control Overview and development of embodied AI, implementation of PID control algorithms, fundamentals of robot kinematics, Preliminary Embodied Navigation (video tutorial). Hot topic: Reproduction of Spring Festival Gala robots [ /every-embodied/blob/main/07-robot-manipulation,motion-control/Locomotion/01-reproduction-of-dance-robot-in-the-SF-Gala.md Video tutorial: [http://xhslink.com/o/1lB9dX0Vt2t Course 01-Overview of Embodied AI: This chapter covers “Overview and Development of Embodied AI”. [ /datawhalechina/every-embodied/blob/main/01-Overview-of-embodied-ai/01-overview-of-embodied-ai.md Course 02-Basics of robots and control, hand-eye coordination: This chapter covers “Robotics Basics” (kinematics) and “Control” (PID algorithms). [ /datawhalechina/every-embodied/blob/main/02-basics-of-robots-and-control,hand-eye-coordination/01-robot-space-description-and-coordinate-transformation.md /datawhalechina/every-embodied/blob/main/02-basics-of-robots-and-control,hand-eye-coordination/supplemental-code-tutorial-Cartpole-three-control-algorithms-practice/cartpole-modeling-and-control.md **Basics of Embodied Navigation (Priority: Complete assignment): [ /every-embodied/blob/main/08-embodied-navigation-and-VLN/02-basics-of-simulation-environment/habitat-navigation-environment/habitat_lab-environment-building-and-configuration.md /every-embodied/blob/main/08-embodied-navigation-and-VLN/02-basics-of-simulation-environment/habitat-navigation-environment/habitat_sim-basic-practice.md /datawhalechina/every-embodied/blob/main/08-embodied-navigation-and-VLN/02-basics-of-simulation-environment/habitat-navigation-environment/habitat_lab-basic.md **Video tutorial: [http://xhslink.com/o/4FGHWn2YICu |
4 days | Deadline: April 17, 03:00 | ||
| Task02: Vision Perception and Hand-Eye Coordination Application of computer vision in robots, implementation of hand-eye calibration algorithms, depth estimation and 3D reconstruction. [ Course 02-Basics of robots and control, hand-eye coordination: This chapter covers “Robotics Basics” (kinematics) and “Control” (PID algorithms). [ /every-embodied/blob/main/02-basics-of-robots-and-control,hand-eye-coordination/05-combined-practice-and-simulation.md Course 04-Computer vision and 3D reconstruction in embodied scenarios: This chapter covers “Computer vision” and “3D reconstruction/depth estimation”. [ /datawhalechina/every-embodied/tree/main/04-computer-vision-in-embedded-scenarios-3D-reconstruction |
4 days | Deadline: April 21, 03:00 | ||
| Task03: Reinforcement Learning and Intelligent Decision-Making Reinforcement learning framework for robots, application of PPO algorithm in robot control, GenieSim simulator (priority: learn, with video tutorial), Maniskill environment building and training. [ Course 05-Depth and reinforcement learning in embodied scenarios: This chapter covers “Reinforcement Learning and Intelligent Decision-Making” and “PPO algorithm”. [ /every-embodied/blob/main/05-depth-and-reinforcement-learning-in-embedded-scenarios/01-multi-robot-furniture-carrying-reinforcement-learning.md **GenieSim simulator (priority: complete assignment): [ Video: [http://xhslink.com/o/2tybo79bOsk Text: /every-embodied/blob/main/10-other-embedded-ai-simulation-tools-and-simulation-frontiers/08-GenieSim3-configuration.md Course 07-Robot simulation tools and benchmarks: This chapter covers Maniskill (optional). [ /every-embodied/blob/main/10-other-embedded-ai-simulation-tools-and-simulation-frontiers/02-Maniskill-environment-simulation-configuration.md |
4 days | Deadline: April 25, 03:00 | ||
| Task04: Hardware Practice and Cutting-Edge Technologies Control of DK-X5 development board, LeRobot VLA control, mujoco grasping experiments (video tutorial). [ Course 03-Robotics hardware and sweetpotato RDK-X5 development board control tutorial: This chapter covers “Control of DK-X5 development board”. Optional for those with a development board. [ /every-embodied/blob/main/03-robotics-hardware,lerobot-and-sweetpotato-RDK-X5-development-board-control-tutorial/01-RDKX5-beginner-tutorial.md Course 08-Massive model control, VLA, VLM: This chapter covers LeRobot VLA, a cutting-edge application of vision-language-action model (VLA). [ /every-embodied/blob/main/06-policy-grasping-or-grasping-VLA/massive-model-control,VLA,VLM/01-SmolVLA-LIBERO/01-SmolVLA-libero.md **Priority: Complete Assignment 1: Reproduction of ETPNav cutting-edge VLN navigation algorithm (video tutorial) [ http://xhslink.com/o/8t08X6dROt5 **Priority: Complete Assignment 2: mujoco grasping experiments (video tutorial) [ https://www.bilibili.com/video/BV1WhxeznE61/ |
4 days | Deadline: April 29, 03:00 | ||
| Task05：Frontier Embodied AI Project Reproduction ** UAV Multimodal Practice (Prioritize completing the task) /every-embodied/blob/main/13-Other Frontier Projects Reproduction/UAV Large Model %2BGroundingdino Practice/UAV Multimodal Large Model.md Corresponding simulation器 download link (Windows): https://huggingface.co/datasets/Datawhale/house_win ACT and Pi0 Data Collection and Algorithm Practice (Prioritize completing the task)** /every-embodied/blob/main/06-Policy Grabbing or Grabbing VLA/Large Model Control, VLA, VLM/04mujoco Reproduction ACT, Pi0, SmolVLA/1.collect_data.ipynb Video tutorial: [http://xhslink.com/o/8lsptSLOlex] |
4 days | Deadline: May 03, 03:00 |