Ai2 achieved this by getting human annotators to explain the pictures within the model’s training data set in excruciating detail over multiple pages of text. They asked the annotators to discuss what they saw as an alternative of typing it. Then they used AI techniques to convert their speech into data, which made the training process much quicker while reducing the computing power required.
These techniques could prove really useful if we would like to meaningfully govern the information that we use for AI development, says Yacine Jernite, who’s the machine learning and society lead at Hugging Face, and was not involved within the research.
“It is smart that on the whole, training on higher-quality data can lower the compute costs,” says Percy Liang, the director of the Stanford Center for Research on Foundation Models, who also didn’t take part in the research.
One other impressive capability is that the model can “point” at things, meaning it could analyze elements of a picture by identifying the pixels that answer queries.
In a demo shared with , Ai2 researchers took a photograph outside their office of the local Seattle marina and asked the model to discover various elements of the image, similar to deck chairs. The model successfully described what the image contained, counted the deck chairs, and accurately pinpointed to other things within the image because the researchers asked. It was not perfect, nonetheless. It couldn’t locate a particular car parking zone, for instance.
