In last week’s demo, Raul Puri, a scientist who works on GPT-4, gave me a fast tour of the image recognition feature. He uploaded a photograph of a child’s math homework, circled a Sudoku-like puzzle on the screen, and asked ChatGPT the way you were meant to resolve it. ChatGPT replied with the right steps.
Puri says he has also used the feature to assist him fix his fiancée’s computer by uploading screenshots of error messages and asking ChatGPT what he should do. “This was a really painful experience that it helped me get through,” he says.
ChatGPT’s image recognition ability has already been trialed by an organization called Be My Eyes, which makes an app for individuals with impaired vision. Users can upload a photograph of what’s in front of them and ask human volunteers to inform them what it’s. In a partnership with OpenAI, Be My Eyes gives its users the choice of asking a chatbot as an alternative.
“Sometimes my kitchen is just a little messy, or it’s just very early Monday morning and I don’t need to confer with a human being,” Be My Eyes founder Hans Jørgen Wiberg, who uses the app himself, told me once I interviewed him at EmTech Digital in May. “Now you possibly can ask the photo questions.”
OpenAI is aware of the chance of releasing these updates to the general public. Combining models brings whole latest levels of complexity, says Puri. He says his team has spent months brainstorming possible misuses. You can’t ask questions on photos of personal individuals, for instance.
Jang gives one other example: “Straight away should you ask ChatGPT to make a bomb it would refuse,” she says. “But as an alternative of claiming, ‘Hey, tell me find out how to make a bomb,’ what should you showed it a picture of a bomb and said, ‘Are you able to tell me find out how to make this?’”
“You may have all the issues with computer vision; you will have all the issues of huge language models. Voice fraud is a giant problem,” says Puri. “You may have to contemplate not only our users, but in addition the those who aren’t using the product.”