While the aim of the study was to not prove that AI systems are able to replacing humans in creative roles, it raises philosophical questions on the characteristics which are unique to humans, says Simone Grassini, an associate professor of psychology on the University of Bergen, Norway, who co-led the research.

“We’ve shown that previously few years, technology has taken a really big breakthrough after we discuss imitating human behavior,” he says. “These models are constantly evolving.” 

Proving that machines can perform well in tasks designed for measuring creativity in humans doesn’t exhibit that they’re able to anything approaching original thought, says Ryan Burnell, a senior research associate on the Alan Turing Institute, who was not involved with the research.

The chatbots that were tested are “black boxes,” meaning that we don’t know exactly what data they were trained on, or how they generate their responses, he says. “What’s very plausibly happening here is that a model wasn’t coming up with latest creative ideas—it was just drawing on things it’s seen in its training data, which could include this exact Alternate Uses Task,” he explains. “In that case, we’re not measuring creativity. We’re measuring the model’s past knowledge of this sort of task.”

That doesn’t mean that it’s not still useful to match how machines and humans approach certain problems, says Anna Ivanova, an MIT postdoctoral researcher studying language models, who didn’t work on the project. 

Nonetheless, we should always keep in mind that although chatbots are excellent at completing specific requests, slight tweaks like rephrasing a prompt could be enough to stop them from performing as well, she says. Ivanova believes that these sorts of studies should prompt us to look at the link between the duty we’re asking AI models to finish and the cognitive capability we’re attempting to measure. “We shouldn’t assume that folks and models solve problems in the identical way,” she says.