Post-Training Isaac GR00T N1.5 for LeRobot SO-101 Arm

NVIDIA Isaac GR00T (Generalist Robot 00 Technology) is a research and development platform for constructing robot foundation models and data pipelines, designed to speed up the creation of intelligent, adaptable robots.

Today, we announced the supply of Isaac GR00T N1.5, the primary major update to Isaac GR00T N1, the world’s first open foundation model for generalized humanoid robot reasoning and skills. This cross-embodiment model processes multimodal inputs, including language and pictures, to perform manipulation tasks across diverse environments. It’s adaptable through post-training for specific embodiments, tasks, and environments.

On this blog, we’ll reveal learn how to post-train (fine-tune) GR00T N1.5 using teleoperation data from a single SO-101 robot arm.

Technical Blog for GR00T N1.5: https://research.nvidia.com/labs/gear/gr00t-n1_5/

Step-by-step tutorial

Now accessible to developers working with a wide selection of robot form aspects, GR00T N1.5 will be easily fine-tuned and adapted using the reasonably priced, open-source LeRobot SO-101 arm.

This flexibility is enabled by the EmbodimentTag system, which allows seamless customization of the model for various robotic platforms, empowering hobbyists, researchers, and engineers to tailor advanced humanoid reasoning and manipulation capabilities to their very own hardware.

Step 0: Installation

Before proceeding to installation, please check if you happen to satisfy the prerequisites.

0.1 Clone the Isaac-GR00T Repo

git clone https://github.com/NVIDIA/Isaac-GR00T
cd Isaac-GR00T

0.2 Create the environment

conda create -n gr00t python=3.10
conda activate gr00t
pip install --upgrade setuptools
pip install -e .[base]
pip install --no-build-isolation flash-attn==2.7.1.post4 

Step 1: Dataset Preparation

Users can fine-tune GROOT N1.5 with any LeRobot dataset. For this tutorial, we can be using the table cleanup task for instance for fine-tuning.

It is vital to notice that datasets for the SO-100 or SO-101 should not in GR00T N1.5’s initial pre-training. In consequence of this, we can be training it as a new_embodiment.

1.1 Create or Download Your Dataset

For this tutorial, you possibly can either begin by creating your personal custom dataset by following these instructions (really useful) or by downloading the so101-table-cleanup dataset from Hugging Face. The --local-dir argument specifies where the dataset can be saved in your machine.

huggingface-cli download 
    --repo-type dataset youliangtan/so101-table-cleanup 
    --local-dir ./demo_data/so101-table-cleanup

1.2 Configure Modality File

The modality.json file provides additional information in regards to the state and motion modalities to make it “GR00T-compatible”. Copy over the getting_started/examples/so100_dualcam__modality.json to the dataset /meta/modality.json using this command:

cp getting_started/examples/so100_dualcam__modality.json ./demo_data/so101-table-cleanup/meta/modality.json

NOTE: For a single-camera setup like
so100_strawberry_grape, run:

cp getting_started/examples/so100__modality.json ./demo_data//meta/modality.json

After these steps, the dataset will be loaded using the GR00T LeRobotSingleDataset class. An example script for loading the dataset is shown here:

python scripts/load_dataset.py --dataset-path ./demo_data/so101-table-cleanup --plot-state-action --video-backend torchvision_av

Step 2: Positive-tuning the Model

Positive-tuning GR00T N1.5 will be executed using the Python script, scripts/gr00t_finetune.py. To start finetuning, execute the next command out of your terminal:

python scripts/gr00t_finetune.py 
   --dataset-path ./demo_data/so101-table-cleanup/ 
   --num-gpus 1 
   --output-dir ./so101-checkpoints  
   --max-steps 10000 
   --data-config so100_dualcam 
   --video-backend torchvision_av

Tip: The default fine-tuning settings require ~25G of VRAM. Should you do not have that much VRAM, try adding the --no-tune_diffusion_model flag to the gr00t_finetune.py script.

Step 3: Open-loop Evaluation

Once the training is complete and your fine-tuned policy is generated, you possibly can visualize its performance in an open-loop setting by running the next command:

python scripts/eval_policy.py --plot 
   --embodiment_tag new_embodiment 
   --model_path ./so101-checkpoints 
   --data_config so100_dualcam 
   --dataset_path ./demo_data/so101-table-cleanup/ 
   --video_backend torchvision_av 
   --modality_keys single_arm gripper

Congratulations! You’ve successfully finetuned GR00T-N1.5 on a brand new embodiment.

Step 4: Deployment

After successfully fine-tuning and evaluating your policy, the ultimate step is to deploy it onto your physical robot for real-world execution.

To attach your SO-101 robot and start the evaluation, execute the next commands in your terminal:

1. First Run the Policy as a server:

python scripts/inference_service.py --server 
    --model_path ./so101-checkpoints 
    --embodiment-tag new_embodiment 
    --data-config so100_dualcam 
    --denoising-steps 4

2. On a separate terminal, run the eval script as client. Be sure to update the port and id for the robot, in addition to the index and parameters for cameras, to match your configuration.

python getting_started/examples/eval_lerobot.py 
    --robot.type=so100_follower 
    --robot.port=/dev/ttyACM0 
    --robot.id=my_awesome_follower_arm 
    --robot.cameras="{ wrist: {type: opencv, index_or_path: 9, width: 640, height: 480, fps: 30}, front: {type: opencv, index_or_path: 15, width: 640, height: 480, fps: 30}}" 
    --policy_host=10.112.209.136 
    --lang_instruction="Grab pens and place into pen holder."

Since we finetuned GRO0T-N1.5 with different language instructions, the user can steer the policy through the use of certainly one of the duty prompts in the dataset akin to:

“Grab tapes and place into pen holder”.

For detailed step-by-step tutorial, please take a look at: https://github.com/NVIDIA/Isaac-GR00T/tree/fundamental/getting_started

🎉 Glad Hacking! 💻🛠️

Get Began Today

Able to elevate your robotics projects with NVIDIA’s GR00T N1.5? Dive in with these essential resources:

• GR00T N1.5 Model: Download the newest models directly from Hugging Face.
• Positive-Tuning Resources: Find sample datasets and PyTorch scripts for fine-tuning on our GitHub.
• Contribute Datasets: Empower the robotics community by contributing your personal datasets to Hugging Face.
• LeRobot Hackathon: Join the worldwide community and take part in the upcoming LeRobot hackathon to use your skills.

Stay awake-to-date with the newest developments by following NVIDIA on Hugging Face.