logo Mr.Robota 具身智能学习手册
/
中文
Chapter 10: Reproduction of RL Training for Heterogeneous Robots with UniLab + MotrixSim

🤖 NVIDIA Isaac GR00T

📋 Table of Contents

📖 Introduction

NVIDIA Isaac GR00T (Generalist Robot 00 Technology) is a general-purpose base model and development platform designed for humanoid robots by NVIDIA. As a research and development platform, GR00T aims to accelerate the research and development of humanoid robots, enabling them to understand natural language commands and mimic actions by observing human movements. This allows robots to quickly learn coordination, dexterity, and other skills.

GR00T is an important step toward Artificial General Robotics. It combines multimodal perception, reasoning, and control capabilities, enabling robots to navigate, adapt, and interact in the real world.

🔧 Technical Architecture

The core technical architecture of Isaac GR00T includes the following key components:

1. Dual-System Cognitive Architecture

GR00T adopts a dual-system architecture inspired by human cognitive principles:

  • System 1 (fast thinking): Similar to the rapid thinking mechanism of human reflexes or intuition, it is responsible for transforming plans into precise and continuous robotic actions.
  • System 2 (slow thinking): Used for deliberate and organized decision-making. It can reason about the environment and received instructions, thereby planning actions.

2. Multimodal Input Processing

GR00T can handle input in various modalities, including:

  • Natural language instructions
  • Visual information (images and videos)
  • Sensor data

3. Simulation and Data Generation Framework

  • NVIDIA Omniverse platform: Used for generating synthetic data
  • Isaac GR00T Blueprint: The blueprint used for synthetic data generation
  • Newton physics engine: An open-source physics engine developed in collaboration with Google DeepMind and Disney Research for robot development

4. Hardware Support

  • Jetson AGX Thor: A new computing platform designed for humanoid robots, based on the NVIDIA Thor system on chip (SoC).
  • It features next-generation GPUs based on the NVIDIA Blackwell architecture, with a transformer engine, and offers 800 teraflops of 8-bit floating-point AI performance.

💡 GR00T N1 model

Isaac GR00T N1 is the world’s first open, fully customizable humanoid robot base model, used for general humanoid robot reasoning and skills. The N1 model has the following features:

Main Features

  • Cross-entity adaptability: Can adapt to different types of robot entities
  • Multimodal input: Includes language and images
  • General manipulation capabilities: Capable of performing manipulation tasks in diverse environments
  • Customizability: Developers and researchers can use real or synthetic data for post-training training of GR00T N1 to adapt it to specific humanoid robots or tasks

Training Data

GR00T N1 was trained on a large human-shaped dataset, which includes:

  • Real capture data
  • Synthetic data generated using the NVIDIA Isaac GR00T Blueprint component
  • Video data on the scale of the internet

Skills and Abilities

GR00T N1 can easily perform general processing for common tasks, such as:

  • Grabbing objects
  • Moving objects with one or two arms
  • Transferring items from one arm to another
  • Performing multi-step tasks that require long context and a combination of general skills

These capabilities can be applied in various scenarios, such as material handling, packaging, and inspection.

🌟 Application Scenarios

Isaac GR00T can be applied in various industries and scenarios, including:

Industrial Applications

  • Material handling: Material transport and sorting at warehouses, factories, and logistics centers
  • Packaging: Automated product packaging processes
  • Quality inspection: Product defect detection and quality control

Service Applications

  • Household assistant: Tasks such as organizing and cleaning in a home environment
  • Retail service: Shelf organization and customer service in stores
  • Medical assistance: Simple medical tasks and patient care support

Research and Development

  • Robotics Learning Research: Provides researchers with advanced humanoid robot platforms.
  • Human-Machine Interaction Research: Studies more natural and intuitive human-machine interaction methods.
  • Multimodal AI Research: Explores the collaborative processing of vision, language, and actions.

📥 Installation and Configuration

To start using Isaac GR00T, follow these steps:

System Requirements

  • Hardware:
    • NVIDIA GPU: RTX 3090 or higher recommended
    • Memory: At least 32GB RAM
    • Storage: 50GB+ available space (SSD recommended)
  • Software:
    • Ubuntu 20.04/22.04 LTS or Windows 10/11
    • CUDA 12.0+
    • Python 3.8+

Obtain GR00T N1 model

  1. Access on Hugging Face to download the GR00T N1 2B model.

    bashgit lfs install
git clone https://huggingface.co/nvidia/isaac-gr00t-n1-2b

  2. Or use the Hugging Face API:

    pythonfrom huggingface_hub import snapshot_download

snapshot_download(repo_id="nvidia/isaac-gr00t-n1-2b", local_dir="./isaac-gr00t-n1-2b")

Install dependencies

  1. Create a virtual environment:

    bashconda create -n isaac-groot python=3.9
conda activate isaac-groot

  2. Install necessary dependencies:

    bashpip install torch torchvision torchaudio
pip install omniverse-isaac-sim
pip install transformers accelerate safetensors

🚀 Usage Guide

Basic Usage Process

  1. Import model:

    pythonfrom transformers import AutoModelForCausalLM, AutoTokenizer

model_path = "./isaac-gr00t-n1-2b"
model = AutoModelForCausalLM.from_pretrained(model_path, device_map="auto", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(model_path)

  2. Reasoning using the model:

    python# 示例：接收一个包含图像和文本的输入
inputs = tokenizer("Pick up the red cube and place it in the blue bin", return_tensors="pt").to(model.device)

# 生成操作序列
outputs = model.generate(
    inputs.input_ids,
    max_length=200,
    temperature=0.7,
    top_p=0.9,
)

# 解码输出
result = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(result)

  3. Integration with Isaac Sim:

    python# 假设已经在Isaac Sim中设置了场景和机器人
from omni.isaac.core import World

# 创建模拟世界
world = World()

# 加载模型生成的动作序列
# ...

# 应用到机器人
# ...

# 运行模拟
world.play()

Fine-tuning for specific tasks

  1. Prepare training data: Collect or generate training data suitable for a specific task

  2. Setting fine-tuning process:

    pythonfrom transformers import Trainer, TrainingArguments

training_args = TrainingArguments(
    output_dir="./results",
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=5e-5,
    num_train_epochs=3,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=your_dataset,
)

trainer.train()

  3. Save the fine-tuned model:

    pythonmodel.save_pretrained("./my_tuned_gr00t_model")
tokenizer.save_pretrained("./my_tuned_gr00t_model")

❓ Common Questions

1. What is the difference between GR00T and other robot models?

GR00T is a general-purpose robot base model, rather than a specialized model optimized for specific tasks. It adopts a dual-system architecture that combines a fast system (intuition) and a slow system (reasoning), similar to the human cognitive process. Additionally, GR00T is designed to work across different robot platforms, not limited to specific hardware.

2. What hardware do I need to run GR00T?

Running the full version of GR00T requires a powerful GPU support. For complete inference and training, NVIDIA RTX 3090 or a higher-end GPU is recommended. For lightweight applications, cloud services or the NVIDIA Jetson platform can be used. The Jetson AGX Thor, designed specifically for humanoid robots, is ideal hardware for running GR00T.

3. How to obtain the training GR00T data?

NVIDIA has released the GR00T N1 dataset as part of a larger open-source physical AI dataset, which can be downloaded from Hugging Face and GitHub. Additionally, NVIDIA Isaac GR00T Blueprint provides a framework for generating synthetic manipulation actions, helping to create your own training data.

4. With which robots can GR00T be used?

GR00T is designed for use with various humanoid robots, including those from 1X Technologies, Agility Robotics, Apptronik, Boston Dynamics, and Figure AI. Due to its versatility, GR00T can adapt to different robot forms through appropriate post-training adjustments.

📚 Resources and References

This guide provides a basic introduction and usage instructions for NVIDIA Isaac GR00T. As technology continues to evolve and improve, it is recommended to regularly check the NVIDIA official documentation for the latest information.