From wiping up spills to serving up food, robots are being taught to perform increasingly complicated household tasks. Many such home-bot trainees are learning through imitation; they’re programmed to repeat the motions that a human physically guides them through.

It seems that robots are excellent mimics. But unless engineers also program them to regulate to each possible bump and nudge, robots don’t necessarily know the best way to handle these situations, in need of starting their task from the highest.

Now MIT engineers are aiming to present robots a little bit of common sense when faced with situations that push them off their trained path. They’ve developed a technique that connects robot motion data with the “common sense knowledge” of enormous language models, or LLMs.

Their approach enables a robot to logically parse many given household task into subtasks, and to physically adjust to disruptions inside a subtask in order that the robot can move on without having to return and begin a task from scratch — and without engineers having to explicitly program fixes for each possible failure along the way in which.   

“Imitation learning is a mainstream approach enabling household robots. But when a robot is blindly mimicking a human’s motion trajectories, tiny errors can accumulate and eventually derail the remaining of the execution,” says Yanwei Wang, a graduate student in MIT’s Department of Electrical Engineering and Computer Science (EECS). “With our method, a robot can self-correct execution errors and improve overall task success.”

Wang and his colleagues detail their recent approach in a study they are going to present on the International Conference on Learning Representations (ICLR) in May. The study’s co-authors include EECS graduate students Tsun-Hsuan Wang and Jiayuan Mao, Michael Hagenow, a postdoc in MIT’s Department of Aeronautics and Astronautics (AeroAstro), and Julie Shah, the H.N. Slater Professor in Aeronautics and Astronautics at MIT.

Language task

The researchers illustrate their recent approach with a straightforward chore: scooping marbles from one bowl and pouring them into one other. To perform this task, engineers would typically move a robot through the motions of scooping and pouring — multi function fluid trajectory. They could do that multiple times, to present the robot a variety of human demonstrations to mimic.

“However the human demonstration is one long, continuous trajectory,” Wang says.

The team realized that, while a human might show a single task in a single go, that task is determined by a sequence of subtasks, or trajectories. As an example, the robot has to first reach right into a bowl before it will possibly scoop, and it must scoop up marbles before moving to the empty bowl, and so forth. If a robot is pushed or nudged to make a mistake during any of those subtasks, its only recourse is to stop and begin from the start, unless engineers were to explicitly label each subtask and program or collect recent demonstrations for the robot to recuperate from the said failure, to enable a robot to self-correct within the moment.

“That level of planning could be very tedious,” Wang says.

As a substitute, he and his colleagues found a few of this work may very well be done mechanically by LLMs. These deep learning models process immense libraries of text, which they use to ascertain connections between words, sentences, and paragraphs. Through these connections, an LLM can then generate recent sentences based on what it has learned concerning the sort of word that’s prone to follow the last.

For his or her part, the researchers found that along with sentences and paragraphs, an LLM may be prompted to supply a logical list of subtasks that may be involved in a given task. As an example, if queried to list the actions involved in scooping marbles from one bowl into one other, an LLM might produce a sequence of verbs reminiscent of “reach,” “scoop,” “transport,” and “pour.”

“LLMs have a solution to let you know the best way to do each step of a task, in natural language. A human’s continuous demonstration is the embodiment of those steps, in physical space,” Wang says. “And we wanted to attach the 2, in order that a robot would mechanically know what stage it’s in a task, and have the opportunity to replan and recuperate by itself.”

Mapping marbles

For his or her recent approach, the team developed an algorithm to mechanically connect an LLM’s natural language label for a selected subtask with a robot’s position in physical space or a picture that encodes the robot state. Mapping a robot’s physical coordinates, or a picture of the robot state, to a natural language label is referred to as “grounding.” The team’s recent algorithm is designed to learn a grounding “classifier,” meaning that it learns to mechanically discover what semantic subtask a robot is in — for instance, “reach” versus “scoop” — given its physical coordinates or a picture view.

“The grounding classifier facilitates this dialogue between what the robot is doing within the physical space and what the LLM knows concerning the subtasks, and the constraints you will have to listen to inside each subtask,” Wang explains.

The team demonstrated the approach in experiments with a robotic arm that they trained on a marble-scooping task. Experimenters trained the robot by physically guiding it through the duty of first reaching right into a bowl, scooping up marbles, transporting them over an empty bowl, and pouring them in. After a couple of demonstrations, the team then used a pretrained LLM and asked the model to list the steps involved in scooping marbles from one bowl to a different. The researchers then used their recent algorithm to attach the LLM’s defined subtasks with the robot’s motion trajectory data. The algorithm mechanically learned to map the robot’s physical coordinates within the trajectories and the corresponding image view to a given subtask.

The team then let the robot perform the scooping task by itself, using the newly learned grounding classifiers. Because the robot moved through the steps of the duty, the experimenters pushed and nudged the bot off its path, and knocked marbles off its spoon at various points. Relatively than stop and begin from the start again, or proceed blindly with no marbles on its spoon, the bot was capable of self-correct, and accomplished each subtask before moving on to the subsequent. (As an example, it could be certain that it successfully scooped marbles before transporting them to the empty bowl.)

“With our method, when the robot is making mistakes, we don’t have to ask humans to program or give extra demonstrations of the best way to recuperate from failures,” Wang says. “That’s super exciting because there’s an enormous effort now toward training household robots with data collected on teleoperation systems. Our algorithm can now convert that training data into robust robot behavior that may do complex tasks, despite external perturbations.”