While other forms of AI, comparable to large language models, are trained on huge repositories of information scraped from the web, the identical can’t be done with robots, because the info must be physically collected. This makes it so much harder to construct and scale training databases.  

Similarly, while it’s relatively easy to coach robots to execute tasks inside a laboratory, these conditions don’t necessarily translate to the messy unpredictability of an actual home. 

To combat these problems, the team got here up with an easy, easily replicable solution to collect the info needed to coach Dobb-E—using an iPhone attached to a reacher-grabber stick, the type typically used to choose up trash. Then they set the iPhone to record videos of what was happening.

Volunteers in 22 homes in Latest York accomplished certain tasks using the stick, including opening and shutting doors and drawers, turning lights on and off, and placing tissues within the trash. The iPhones’ lidar systems, motion sensors, and gyroscopes were used to record data on movement, depth, and rotation—necessary information with regards to training a robot to copy the actions by itself.

After they’d collected just 13 hours’ value of recordings in total, the team used the info to coach an AI model to instruct a robot in the right way to perform the actions. The model used self-supervised learning techniques, which teach neural networks to identify patterns in data sets by themselves, without being guided by labeled examples.

The subsequent step involved testing how reliably a commercially available robot called Stretch, which consists of a wheeled unit, a tall pole, and a retractable arm, was in a position to use the AI system to execute the tasks. An iPhone held in a 3D-printed mount was attached to Stretch’s arm to copy the setup on the stick.

The researchers tested the robot in 10 homes in Latest York over 30 days, and it accomplished 109 household tasks with an overall success rate of 81%. Each task typically took Dobb-E around 20 minutes to learn: five minutes of demonstration from a human using the stick and attached iPhone, followed by quarter-hour of fine-tuning, when the system compared its previous training with the brand new demonstration.