In-depth Understanding

In this section, we will delve into the training configuration options. We will also explain in detail aspects such as implementation labels, modality configurations, and data transformation.

Implement minor head adjustment

GR00T features a dedicated action head design, allowing it to work with various types of robots. During fine-tuning, the implementation heads to be trained must be specified based on the dataset:

1. Implementation Tag

   • When instantiating the LeRobotSingleDataset class, each dataset must be marked with a specific EmbodimentTag (e.g., EmbodimentTag.GR1_UNIFIED)
   • A detailed list of implementation tags can be found in gr00t/data/embodiment_tags.py
   • This tag determines which action head will be fine-tuned
   • If there is a new implementation, the EmbodimentTag.NEW_EMBODIMENT tag can be used (e.g., new_embodiment.your_custom_dataset)

2. Working Principle

   • When a dataset is loaded using a specific implementation tag (e.g., EmbodimentTag.GR1_UNIFIED)
   • The model has multiple components that can be configured for fine-tuning (visual encoder, language model, DiT, etc.)
   • Especially for the action heads, only the heads corresponding to the specified implementation tags are fine-tuned. The other action heads of specific implementations remain frozen

Advanced Adjustment Parameters

Model Components

This model has several components that can be adjusted independently. These parameters can be configured in the GR00T_N1.from_pretrained function.

1. Visual encoder (tune_visual)

   • If the data has visual characteristics different from pre-trained data, set it to true
   • Note: This is computationally expensive
   • Default value: false

2. Language model (tune_llm)

   • Set to true only when there is a language from a specific domain that differs significantly from standard instructions.
   • In most cases, it should be false.
   • Default value: false

3. Projector (tune_projector)

   • By default, the projector is adjusted.
   • This helps align the action and state spaces of specific implementations.

4. Diffusion model（tune_diffusion_model

   • By default, the diffusion model is not adjusted
   • This is the action head shared by all implementers of the projector

Understanding Data Conversion

This document explains the different types of transformations used in our data processing pipeline. There are four main categories of transformations:

1. Video Conversion

Video conversion is applied to video data to prepare for model training. According to our experimental evaluation, the following video conversion combinations have the best effect:

• VideoToTensor: Converts video data from its original format into a PyTorch tensor for processing.
• VideoCrop: Crops video frames, using a 0.95 ratio factor in random mode to introduce minor variations.
• VideoResize: Adjusts video frames to a standard size (224x224 pixels) using linear interpolation.
• VideoColorJitter: Applies color enhancement by randomly adjusting brightness (±0.3), contrast (±0.4), saturation (±0.5), and hue (±0.08).
• VideoToNumpy: Converts the processed tensor back to a NumPy array for further processing.

2. State Transition

Process robot state information for status transition:

• StateActionToTensor: Converts state data (such as arm position and hand configuration) into PyTorch tensors.
• StateActionTransform: Applies normalization to the state data. Different normalization modes are available based on the modal key. The conversion logic can be found in state_action.py file.

3. Action Conversion

Action conversion processing of robot action data:

• StateActionToTensor: Similar to state transformation, it converts action data into PyTorch tensors.
• StateActionTransform: Applies normalization to action data. Like state data, min-max normalization is used to standardize the action values for the left/right arms, hands, and waist.

4. Connection Conversion

ConcatTransform combines the processed data into a unified array:

• It connects video data according to the specified video modal key order.
• It connects state data according to the specified state modal key order.
• It connects action data according to the specified action modal key order.

This connection step is crucial, as it prepares the data in the format expected by the model, ensuring all modalities are properly aligned and ready for training or inference.

5. GR00T Conversion

GR00TTransform is a custom transformation used to prepare data for the model. It is applied at the end of the data pipeline.

• It fills the data into the maximum length of the sequence in the batch.
• It creates a dictionary, using the modality key as a key and the processed data as a value.

In practice, usually no extensive modifications are needed to this conversion; if any are required.

Lerobot Dataset Compatibility

More details about the GR00T-compatible lerobot dataset can be found in LeRobot_compatible_data_schema.md.