Researchers used the system, called LucidSim, to coach a robot dog in parkour, getting it to scramble over a box and climb stairs though it had never seen any real-world data. The approach demonstrates how helpful generative AI could possibly be with regards to teaching robots to do difficult tasks. It also raises the chance that we could ultimately train them in entirely virtual worlds. The research was presented on the Conference on Robot Learning (CoRL) last week.
“We’re in the midst of an industrial revolution for robotics,” says Ge Yang, a postdoc at MIT’s Computer Science and Artificial Intelligence Laboratory, who worked on the project. “That is our attempt at understanding the impact of those [generative AI] models outside of their original intended purposes, with the hope that it is going to lead us to the subsequent generation of tools and models.”
LucidSim uses a mixture of generative AI models to create the visual training data. First the researchers generated hundreds of prompts for ChatGPT, getting it to create descriptions of a spread of environments that represent the conditions the robot would encounter in the actual world, including several types of weather, times of day, and lighting conditions. These included “an ancient alley lined with tea houses and small, quaint shops, each displaying traditional ornaments and calligraphy” and “the sun illuminates a somewhat unkempt lawn dotted with dry patches.”
These descriptions were fed right into a system that maps 3D geometry and physics data onto AI-generated images, creating short videos mapping a trajectory for the robot to follow. The robot draws on this information to work out the peak, width, and depth of the things it has to navigate—a box or a set of stairs, for instance.
The researchers tested LucidSim by instructing a four-legged robot equipped with a webcam to finish several tasks, including locating a traffic cone or soccer ball, climbing over a box, and walking up and down stairs. The robot performed consistently higher than when it ran a system trained on traditional simulations. In 20 trials to locate the cone, LucidSim had a 100% success rate, versus 70% for systems trained on standard simulations. Similarly, LucidSim reached the soccer ball in one other 20 trials 85% of the time, and just 35% for the opposite system.
Finally, when the robot was running LucidSim, it successfully accomplished all 10 stair-climbing trials, compared with just 50% for the opposite system.
COURTESY OF MIT CSAIL
These results are more likely to improve even further in the longer term if LucidSim draws directly from sophisticated generative video models somewhat than a rigged-together combination of language, image, and physics models, says Phillip Isola, an associate professor at MIT who worked on the research.
The researchers’ approach to using generative AI is a novel one that may pave the best way for more interesting recent research, says Mahi Shafiullah, a PhD student at Recent York University who’s using AI models to coach robots. He didn’t work on the project.
