Researchers at Carnegie Mellon University’s Robotics Institute have developed a tool called FRIDA, which is a robotic arm with a paintbrush attached to it. The tool leverages artificial intelligence (AI) to work along with humans on art projects.

The team is ready to present the research titled “FRIDA: A Collaborative Robot Painter With a Differentiable, Real2Sim2Real Planning Environment” on the 2023 IEEE International Conference on Robotics and Automation in May.

Peter Schaldenbrand is a Ph.D. student within the Robotics Institute on the School of Computer Science. He works with FRIDA and explores AI and creativity.

“There’s this one painting of a frog ballerina that I believe turned out really nicely,” he said. “It is actually silly and fun, and I believe the surprise of what FRIDA generated based on my input was really fun to see.”

FRIDA is an acronym for Framework and Robotics Initiative for Developing Arts. It is called after Frida Kahlo.

The research was led by Schalderbrand, together with RI faculty members Jean Oh and Jim McCaam, and it has enticed students and researchers from throughout CMU.

Collaborative Tool Not Artist

Users can guide FRIDA by inputting a text description, submitting other artistic endeavors to encourage its style, or uploading a photograph and asking it to color a representation of it. The team can be testing other inputs, corresponding to audio.

“FRIDA is a robotic painting system, but FRIDA shouldn’t be an artist,” Schalderbrand continued. “FRIDA shouldn’t be generating the ideas to speak. FRIDA is a system that an artist could collaborate with. The artist can specify high-level goals for FRIDA after which FRIDA can execute them.”

To color a picture, the robot uses AI models which can be comparable to those powering OpenAI’s ChatGPT and DALL-E 2, which produce text or a picture in response to a prompt. FRIDA simulates how it could paint a picture with brush strokes and utilizes machine learning to evaluate its progress as it really works.

The top products of FRIDA are whimsical and impressionistic. The brushstrokes are daring and lack the precision that’s continuously sought in robotic endeavors.

“FRIDA is a project exploring the intersection of human and robotic creativity,” McCann added. “Frida is using the type of AI models which have been developed to do things like caption images and understand scene content and applying it to this artistic generative problem.”

FRIDA uses AI and machine learning several times during its art-making process. First, it spends an hour or more learning the way to use its paintbrush. Then, it employs vision-language models which have been trained on huge datasets pairing text and pictures scraped from the web, corresponding to OpenAI’s Contrastive Language-Image Pre-Training (CLIP), to know the input.

One of the vital significant technical challenges in producing a physical image is reducing the simulation-to-real gap, which is the disparity between what FRIDA creates in simulation and what it paints on the canvas. FRIDA uses an idea often called real2sim2real, where the robot’s actual brush strokes are used to coach the simulator to reflect and mimic the physical capabilities of the robot and painting materials.

FRIDA’s team now goals to handle among the limitations in current large vision-language models by continually refining those they use. They fed the models headlines from news articles to offer them with a way of what was happening on the earth and further trained them on images and text which can be more representative of diverse cultures to avoid an American or Western bias.