We’re thrilled to announce a series of great advancements across LeRobot, designed to make open-source robot learning more powerful, scalable, and user-friendly than ever before! From revamped datasets to versatile editing tools, recent simulation environments, and a groundbreaking plugin system for hardware, LeRobot is constantly evolving to fulfill the demands of cutting-edge embodied AI.
TL;DR
LeRobot v0.4.0 delivers a serious upgrade for open-source robotics, introducing scalable Datasets v3.0, powerful recent VLA models like PI0.5 and GR00T N1.5, and a brand new plugin system for easier hardware integration. The discharge also adds support for LIBERO and Meta-World simulations, simplified multi-GPU training, and a brand new Hugging Face Robot Learning Course.
Table-of-Contents
Datasets: Ready for the Next Wave of Large-Scale Robot Learning
We have completely overhauled our dataset infrastructure with LeRobotDataset v3.0, featuring a brand new chunked episode format and streaming capabilities. This can be a game-changer for handling massive datasets like OXE (Open X Embodiment) and Droid, bringing unparalleled efficiency and scalability.
What’s Recent in Datasets v3.0?
- Chunked Episodes for Massive Scale: Our recent format supports datasets on the OXE-level (> 400GB), enabling unprecedented scalability.
- Efficient Video Storage + Streaming: Enjoy faster loading times and seamless streaming of video data.
- Unified Parquet Metadata: Say goodbye to scattered JSONs! All episode metadata is now stored in unified, structured Parquet files for easier management and access.
- Faster Loading & Higher Performance: Experience significantly reduced dataset initialization times and more efficient memory usage.
We have also provided a conversion script to simply migrate your existing v2.1 datasets to the brand new v3.0 format, ensuring a smooth transition. Read more about it in our previous blog post. Open-source robotics keeps leveling up!
Recent Feature: Dataset Editing Tools!
Working with LeRobot datasets just got an entire lot easier! We have introduced a robust set of utilities for flexible dataset editing.
With our recent lerobot-edit-dataset CLI, you’ll be able to now:
- Delete specific episodes from existing datasets.
- Split datasets by fractions or episode indices.
- Add or remove features with ease.
- Merge multiple datasets into one unified set.
lerobot-edit-dataset
--repo_id lerobot/pusht_merged
--operation.type merge
--operation.repo_ids "['lerobot/pusht_train', 'lerobot/pusht_val']"
lerobot-edit-dataset
--repo_id lerobot/pusht
--new_repo_id lerobot/pusht_after_deletion
--operation.type delete_episodes
--operation.episode_indices "[0, 2, 5]"
These tools streamline your workflow, allowing you to curate and optimize your robot datasets like never before. Take a look at the docs for more details!
Simulation Environments: Expanding Your Training Grounds
We’re constantly expanding LeRobot’s simulation capabilities to supply richer and more diverse training environments on your robotic policies.
LIBERO Support
LeRobot now officially supports LIBERO, certainly one of the biggest open benchmarks for Vision-Language-Motion (VLA) policies, boasting over 130 tasks! This can be a huge step toward constructing the go-to evaluation hub for VLAs, enabling easy integration and a unified setup for evaluating any VLA policy.
Take a look at the LIBERO dataset and our docs to start!
Meta-World Integration
We have integrated Meta-World, a premier benchmark for testing multi-task and generalization abilities in robotic manipulation, featuring over 50 diverse manipulation tasks. This integration, together with our standardized use of gymnasium ≥ 1.0.0 and mujoco ≥ 3.0.0, ensures deterministic seeding and a strong simulation foundation.
Train your policies with the Meta-World dataset today!
Codebase: Powerful Tools For Everyone
We’re making robot control more flexible and accessible, enabling recent possibilities for data collection and model training.
The Recent Pipeline for Data Processing
Getting data from a robot to a model (and back!) is difficult. Raw sensor data, joint positions, and language instructions don’t match what AI models expect. Models need normalized, batched tensors on the suitable device, while your robot hardware needs specific motion commands.
We’re excited to introduce Processors: a brand new, modular pipeline that acts as a universal translator on your data. Consider it as an assembly line where each ProcessorStep handles one specific job—like normalizing, tokenizing text, or moving data to the GPU.
You possibly can chain these steps together into a robust pipeline to perfectly manage your data flow. We have even created two distinct types to make life easier:
PolicyProcessorPipeline: Built for models. It expertly handles batched tensors for high-performance training and inference.RobotProcessorPipeline: Built for hardware. It processes individual data points (like a single remark or motion) for real-time robot control.
obs = robot.get_observation()
obs_processed = preprocess(obs)
motion = model.select_action(obs_processed)
action_processed = postprocess(motion)
robot.send_action(action_processed)
This technique makes it easy to attach any policy to any robot, ensuring your data is all the time in the proper format for each step of the way in which. Learn more about it in our Introduction to Processors documentation.
Multi-GPU Training Made Easy
Training large robot policies just got loads faster! We have integrated Speed up directly into our training pipeline, making it incredibly easy to scale your experiments across multiple GPUs with just one command:
speed up launch
--multi_gpu
--num_processes=$NUM_GPUs
$(which lerobot-train)
--dataset.repo_id=${HF_USER}/my_dataset
--policy.repo_id=${HF_USER}/my_trained_policy
--policy.type=$POLICY_TYPE
Whether you are fine-tuning a policy or running large-scale experiments, LeRobot now handles all of the complexities of distributed training for you. This implies you’ll be able to drastically reduce training time, cutting it in half with 2 GPUs, right down to a 3rd with 3 GPUs, and beyond.
Take a look at the documentation to speed up your robot learning!
Policies: Unleashing Open-World Generalization
PI0 and PI0.5
In a serious milestone for open-source robotics, we have integrated pi0 and pi0.5 policies by Physical Intelligence into LeRobot! These Vision-Language-Motion (VLA) models represent a big leap towards addressing open-world generalization in robotics. But what makes π₀.₅ revolutionary?
- Open-World Generalization: Designed to adapt to thoroughly recent environments and situations, generalizing across physical, semantic, and environmental levels.
- Co-training on Heterogeneous Data: Learns from a various mixture of multimodal web data, verbal instructions, subtask commands, and multi-environment robot data.
- Physical Intelligence Collaboration: Huge due to the Physical Intelligence team for his or her groundbreaking work!
You will discover the models on the Hugging Face Hub: pi0.5_base, pi0_base, and their Libero-tuned counterparts. For more details, checkout the Physical Intelligence Reasearch
GR00T N1.5
In one other exciting development, we have integrated NVIDIA’s GR00T N1.5 into LeRobot, due to a implausible collaboration with the NVIDIA robotics team! This open foundation model is a powerhouse for generalized robot reasoning and skills. As a cross-embodiment model, it takes multimodal input (like language and pictures) to perform complex manipulation tasks in diverse environments, marking one other major leap in generalized robotics. But what makes GR00T N1.5 a game-changer?
- Generalized Reasoning & Skills: Designed as a cross-embodiment foundation model, GR00T N1.5 excels at generalized reasoning and manipulation tasks, with improved language-following ability.
- Expansive Heterogeneous Training: It learns from an enormous dataset combining real captured humanoid data, synthetic data generated by NVIDIA Isaac GR00T Blueprint, and internet-scale video data.
- NVIDIA Collaboration: We’re thrilled to partner with the NVIDIA team to bring this state-of-the-art model to the open-source LeRobot community!
You will discover the model on the Hugging Face Hub: GR00T-N1.5-3B. For more details, take a look at the NVIDIA research page and the official GitHub repository.
The native integration of those policies in lerobot is a big step forward in making robot learning as open and reproducible as it might be. Try them out today, share your runs, and let’s push forward the frontier of embodied AI together!
Robots: A Recent Era of Hardware Integration with the Plugin System
Big news for hardware enthusiasts! We have launched a brand-new plugin system to revolutionize the way you integrate third-party hardware with LeRobot. Now, connecting any robot, camera, or teleoperator is so simple as a pip install, eliminating the necessity to switch the core library.
Key Advantages
- Extensibility: Develop and integrate custom hardware in separate Python packages.
- Scalability: Supports a growing ecosystem of devices without bloating the core library.
- Community-Friendly: Lowers the barrier to entry for community contributions, fostering a more collaborative environment.
Learn the right way to create your personal plugin in our documentation.
pip install lerobot_teleoperator_my_awesome_teleop
lerobot-teleoperate --teleop.type=my_awesome_teleop
Reachy 2 Integration
Due to our recent plugin system, we have also added Reachy 2 from Pollen Robotics to LeRobot! Reachy 2 is on the market for each real robot control and simulation, enabling you to experiment with teleoperation and autonomous demos straight away.
Phone Integration
Due to our powerful recent pipeline system, you’ll be able to now teleoperate your follower arm right out of your phone (iOS/Android). The phone acts as a teleoperator device, and our RobotProcessor pipeline handles all of the transformations, allowing you to drive robots in several motion spaces (like end-effector space) with ease. Take a look at the examples!
The Hugging Face Robot Learning Course
We’re launching a comprehensive, self-paced, and fully open-source course designed to make robot learning accessible to everyone! When you’re interested in how real-world robots learn, that is the proper place to begin.
On this course, you’ll learn the right way to:
- Understand the basics of classical robotics.
- Use generative models for imitation learning (VAEs, diffusion, etc.).
- Apply Reinforcement Learning to real-world robots.
- Explore the newest generalist robot policies like PI0 and SmolVLA.
Join the Hugging Face Robotics organization to follow along and begin your journey!
Deep Dive: The Modern Robot Learning Tutorial
For many who wish to go deeper, we have also published a hands-on tutorial on essentially the most recent advancements in robotics. This guide provides self-contained explanations, re-derives modern techniques from first principles, and includes ready-to-use code examples using LeRobot and Hugging Face.
The tutorial itself is hosted in a Space and it features practical examples using LeRobot, with all models and datasets on the Hugging Hub. It’s also possible to take a look at our paper for an in depth overview.
Final thoughts from the team
Beyond these major features, this release is filled with quite a few bug fixes, documentation improvements, updated dependencies, more examples and higher infrastructure to make your experience with LeRobot smoother and more reliable.
We would like to increase an enormous thanks to everyone locally on your invaluable contributions, feedback, and support. We’re incredibly excited concerning the way forward for open-source robotics and may’t wait to work with you on what’s next!
Stay tuned for more to come back 🤗 Start here!
– The LeRobot team ❤️
