The “ImageNet” of Robotics — When and How?

🧭 TL;DR — Why This Blogpost?

On this post, we:

• Recognize the growing impact of community-contributed LeRobot datasets
• Highlight the present challenges in robotic data collection and curation
• Share practical steps and best practices to maximise the impact of this collective effort
  Our goal is to border generalization as a data problem, and to point out how constructing an open, diverse “ImageNet of robotics” isn’t just possible—but already happening.

Introduction

Recent advances in Vision-Language-Motion (VLA) models have enabled robots to perform a big selection of tasks—from easy commands like “grasp the cube” to more complex activities like folding laundry or cleansing a table. These models aim to realize generalization: the flexibility to perform tasks in novel settings, with unseen objects, and in various conditions.

“The most important challenge in robotics isn’t dexterity, but generalization—across physical, visual, and semantic levels.”
— Physical Intelligence

A robot must “determine easy methods to appropriately perform even a walk in the park in a brand new setting or with latest objects,” and this requires each robust skills and commonsense understanding of the world. Yet, progress is usually limited by the provision of diverse data for such robotic systems.

“Generalization must occur at many levels. On the low level, the robot must understand easy methods to pick up a spoon (by the handle) or plate (by the sting), even when it has not seen these specific spoons or plates before, and even in the event that they are placed in a pile of dirty dishes. At the next level, the robot must understand the semantics of every task—where to place clothes and shoes (ideally within the laundry hamper or closet, not on the bed), and what sort of tool is suitable for wiping down a spill. This generalization requires each robust physical skills and a commonsense understanding of the environment, in order that the robot can generalize at many levels at the identical time, from physical, to visual, to semantic. That is made even harder by the limited availability of diverse data for such robotic systems.”
— Physical Intelligence

From Models to Data: Shifting the Perspective

To simplify, the core of generalist policies lies in an easy idea: co-training on heterogeneous datasets. By exposing VLA models to quite a lot of environments, tasks, and robot embodiments, we are able to teach models not only easy methods to act, but why—easy methods to interpret a scene, understand a goal, and adapt skills across contexts.

💡 “Generalization isn’t only a model property—it’s an information phenomenon.”
It emerges from the variety, quality, and abstraction level of the training data.

This brings us to a fundamental query:

Given current datasets, what’s the upper limit of generalization we are able to expect?

Can a robot meaningfully reply to a very novel prompt—say, *”arrange a surprise party”*—if it has never encountered anything remotely similar during training? Especially when most datasets are collected in academic labs, by a limited number of individuals, under well-controlled setups?

We frame generalization as a data-centric view: treating it because the strategy of abstracting broader patterns from data—essentially “zooming out” to disclose task-agnostic structures and principles. This shift in perspective emphasizes the role of dataset diversity, moderately than model architecture alone, in driving generalization.

Why does Robotics lack its ImageNet Moment?

Up to now, nearly all of robotics datasets come from structured academic environments. Even when we scale as much as tens of millions of demonstrations, one dataset will often dominate, limiting diversity. Unlike ImageNet—which aggregated internet-scale data and captured the true world more holistically—robotics lacks a comparably diverse, community-driven benchmark.

This is essentially because collecting data for robotics requires physical hardware and significant effort.

Constructing a LeRobot Community

That’s why, at LeRobot, we’re working to make robotics data collection more accessible—at home, in school, or anywhere. We’re:

• Simplifying the recording pipeline
• Streamlining uploading to the Hugging Face Hub, to foster community sharing
• Reducing hardware costs

We’re already seeing the outcomes: the variety of community-contributed datasets on the Hub is growing rapidly.

Growth of lerobot datasets on the Hugging Face Hub over time.

If we break down the uploaded datasets by robot type, we see that the majority contributions are to So100 and Koch, making robotic arms and manipulation tasks the first focus of the present LeRobot dataset landscape. Nevertheless, it’s essential to do not forget that the potential reaches far beyond. Domains like autonomous vehicles, assistive robots, and mobile navigation stand to learn just as much from shared data. This momentum brings us closer to a future where datasets reflect a worldwide effort, not only the contributions of a single lab or institution.

Distribution of lerobot datasets by robot type.

Listed below are just a couple of standout community-contributed datasets that show how diverse and imaginative robotics will be:

Explore additional creative datasets under the LeRobot tag on the Hugging Face Hub, and interactively view them within the LeRobot Dataset Visualizer.

Scaling Responsibly

As robotics data collection becomes more democratized, curation becomes the following challenge. While these datasets are still collected in constrained setups, they’re a vital step toward inexpensive, general-purpose robotic policies. Not everyone has access to expensive hardware—but with shared infrastructure and open collaboration, we are able to construct something far greater.

🧠 “Generalization isn’t solved in a lab—it’s taught by the world.”
The more diverse our data, the more capable our models shall be.

Higher data = Higher models

Why does data quality matter? Poor-quality data ends in poor downstream performance, biased outputs, and models that fail to generalize. Hence, efficient and high-quality data collection plays a critical role in advancing generalist robotic policies.

While foundation models in vision and language have thrived on massive, web-scale datasets, robotics lacks an “Web of robots”—an enormous, diverse corpus of real-world interactions. As a substitute, robotic data is fragmented across different embodiments, sensor setups, and control modes, forming isolated data islands.

To beat this, recent approaches like Gr00t organize training data as a pyramid, where:

• Large-scale web and video data form the foundation
• Synthetic data adds simulated diversity
• Real-world robot interactions on the top ground the model in physical execution

Inside this framework, efficient real-world data collection is indispensable—it anchors learned behaviors in actual robotic hardware and closes the sim-to-real gap, ultimately improving the generalization, adaptability, and performance of robotics foundation models.

By expanding the volume and variety of real-world datasets, we reduce fragmentation between heterogeneous data sources. When datasets are disjoint by way of environment, embodiment, or task distribution, models struggle to transfer knowledge across domains.

🔗 Real-world data acts as connective tissue—it aligns abstract priors with grounded motion and enables the model to construct more coherent and transferable representations.

Consequently, increasing the proportion of real robot interactions doesn’t merely enhance realism—it structurally reinforces the links between all layers of the pyramid, resulting in more robust and capable policies.

Data Pyramid for Robot Foundation Model Training. Adapted from Gr00t (Yang et al., 2025). Data quantity decreases while embodiment specificity increases from bottom to top.

Challenges with Current Community Datasets

At LeRobot, we’ve began developing an automatic curation pipeline to post-process community datasets. In the course of the post-processing phase, we’ve identified several areas where improvements can further boost dataset quality and facilitate more practical curation going forward:

1. Incomplete or Inconsistent Task Annotations

Many datasets lack task descriptions, lack details or are ambiguous within the task to be done. Semantics is currently on the core of cognition, meaning that understanding the context and specifics of a task is crucial for robotic performance. Detailed expressions be sure that robots understand exactly what is anticipated, but in addition provide a broader knowledge and vocabulary to the cognition system. Ambiguity can result in incorrect interpretation and, consequently, incorrect actions.

Task instructions will be:

• Empty
• Too short (e.g. “Hold”, “Up”)
• With none specific meaning (e.g. “task desc”, “desc”)

Subtask-level annotations are sometimes missing, making it difficult to model complex task hierarchies.
While this will be handled with VLM, it continues to be higher to have a task annotation provided by the creator of the dataset at hand.

2. Feature Mapping Inconsistencies

Features like images.laptop are ambiguously labeled:

• Sometimes it is a third-person view
• Other times it’s more like a gripper (wrist) camera

Manual mapping of dataset features to standardized names is time-consuming and error-prone.
We will possibly automate feature type inference using VLMs or computer vision models to categorise camera perspectives. Nevertheless, keeping this in mind helps to have a cleaner dataset.

3. Low-Quality or Incomplete Episodes

Some datasets contain:

• Episodes with only one or only a few frames
• Manually deleted data files (e.g., deleted .parquet files without reindexing), breaking the sequential consistency.

4. Inconsistent Motion/State Dimensions

Different datasets use different motion or state dimensions, even for a similar robot (e.g., so100).
Some datasets show inconsistencies in motion/state format.

What Makes a Good Dataset?

Now that we all know that making a high-quality dataset is important for training reliable and generalizable robot policies, we have now outlined a checklist of best practices to help you in collecting effective data.

Image Quality

• ✅ Use preferably two camera views
• ✅ Ensure regular video capture (no shaking)
• ✅ Maintain neutral, stable lighting (avoid overly yellow or blue tones)
• ✅ Ensure consistent exposure and sharp focus
• ✅ Leader arm shouldn’t appear within the frame
• ✅ The only moving objects needs to be the follower arm and the manipulated items (avoid human limbs/bodies)
• ✅ Use a static, non-distracting background, or apply controlled variations
• ✅ Record in high resolution (at the least 480×640 / 720p)

Metadata & Recording Protocol

• ✅ Select the correct robot type within the metadata
  If you happen to’re using a custom robot that is not listed within the official LeRobot config registry,
  we recommend checking how similar robots are named in existing datasets on the LeRobot Hub to make sure consistency.
• ✅ Record videos at roughly 30 frames per second (FPS)
• ✅ If deleting episodes, make certain to update the metadata files accordingly (we’ll provide proper tools to edit datasets)

Feature Naming Conventions

Use a consistent and interpretable naming scheme for all camera views and observations:

Format:

.

Examples:

• images.top
• images.front
• images.left
• images.right

Avoid device-specific names:

• ❌ images.laptop
• ❌ images.phone

For wrist-mounted cameras, specify orientation:

• images.wrist.left
• images.wrist.right
• images.wrist.top
• images.wrist.bottom

Consistent naming improves clarity and helps downstream models higher interpret spatial configurations and multi-view inputs.

Task Annotation

• ✅ Use the task field to clearly describe the robot’s objective
  • Example: Pick the yellow lego block and put it within the box
• ✅ Keep task descriptions concise (between 25–50 characters)
• ✅ Avoid vague or generic names like task1, demo2, etc.

Below, we offer a checklist that serves as a tenet for recording datasets, outlining key points to remember through the data collection process.

Figure 4: Dataset Recording Checklist – a step-by-step guide to make sure consistent and high-quality real-world data collection.

How Can You Help?

The following generation of generalist robots won’t be built by a single person or lab — they’ll be built by all of us. Whether you are a student, a researcher, or simply robot-curious, here’s how you may jump in:

• 🎥 Record your personal datasets — Use LeRobot tools to capture and upload good quality datasets out of your robots.
• 🧠 Improve dataset quality — Follow our checklist, clean up your recordings, and help set latest standards for robotics data.
• 📦 Contribute to the Hub — Upload datasets, share examples, and explore what others are constructing.
• 💬 Join the conversation — Give feedback, request features, or help shape the roadmap by engaging in our LeRobot Discord Server.
• 🌍 Grow the movement — Introduce LeRobot to your club, classroom, or lab. More contributors = higher generalization.

Start recording, start contributing—because the longer term of generalist robots is dependent upon the information we construct today.