Computers possess two remarkable capabilities with respect to pictures: They will each discover them and generate them anew. Historically, these functions have stood separate, akin to the disparate acts of a chef who is sweet at creating dishes (generation), and a connoisseur who is sweet at tasting dishes (recognition).

Yet, one can’t help but wonder: What would it not take to orchestrate a harmonious union between these two distinctive capacities? Each chef and connoisseur share a typical understanding within the taste of the food. Similarly, a unified vision system requires a deep understanding of the visual world.

Now, researchers in MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) have trained a system to infer the missing parts of a picture, a task that requires deep comprehension of the image’s content. In successfully filling within the blanks, the system, generally known as the Masked Generative Encoder (MAGE), achieves two goals at the identical time: accurately identifying images and creating recent ones with striking resemblance to reality. 

This dual-purpose system enables myriad potential applications, like object identification and classification inside images, swift learning from minimal examples, the creation of images under specific conditions like text or class, and enhancing existing images.

Unlike other techniques, MAGE doesn’t work with raw pixels. As an alternative, it converts images into what’s called “semantic tokens,” that are compact, yet abstracted, versions of a picture section. Consider these tokens as mini jigsaw puzzle pieces, each representing a 16×16 patch of the unique image. Just as words form sentences, these tokens create an abstracted version of a picture that may be used for complex processing tasks, while preserving the data in the unique image. Such a tokenization step may be trained inside a self-supervised framework, allowing it to pre-train on large image datasets without labels. 

Now, the magic begins when MAGE uses “masked token modeling.” It randomly hides a few of these tokens, creating an incomplete puzzle, after which trains a neural network to fill within the gaps. This manner, it learns to each understand the patterns in a picture (image recognition) and generate recent ones (image generation).

“One remarkable a part of MAGE is its variable masking strategy during pre-training, allowing it to coach for either task, image generation or recognition, inside the same system,” says Tianhong Li, a PhD student in electrical engineering and computer science at MIT, a CSAIL affiliate, and the lead writer on a paper concerning the research. “MAGE’s ability to work within the ‘token space’ slightly than ‘pixel space’ leads to clear, detailed, and high-quality image generation, in addition to semantically wealthy image representations. This might hopefully pave the best way for advanced and integrated computer vision models.” 

Aside from its ability to generate realistic images from scratch, MAGE also allows for conditional image generation. Users can specify certain criteria for the photographs they need MAGE to generate, and the tool will cook up the suitable image. It’s also able to image editing tasks, akin to removing elements from a picture while maintaining a sensible appearance.

Recognition tasks are one other strong suit for MAGE. With its ability to pre-train on large unlabeled datasets, it will probably classify images using only the learned representations. Furthermore, it excels at few-shot learning, achieving impressive results on large image datasets like ImageNet with only a handful of labeled examples.

The validation of MAGE’s performance has been impressive. On one hand, it set recent records in generating recent images, outperforming previous models with a big improvement. However, MAGE topped in recognition tasks, achieving an 80.9 percent accuracy in linear probing and a 71.9 percent 10-shot accuracy on ImageNet (this implies it appropriately identified images in 71.9 percent of cases where it had only 10 labeled examples from each class).

Despite its strengths, the research team acknowledges that MAGE is a piece in progress. The means of converting images into tokens inevitably results in some loss of knowledge. They’re keen to explore ways to compress images without losing necessary details in future work. The team also intends to check MAGE on larger datasets. Future exploration might include training MAGE on larger unlabeled datasets, potentially resulting in even higher performance. 

“It has been an extended dream to attain image generation and image recognition in a single single system. MAGE is a groundbreaking research which successfully harnesses the synergy of those two tasks and achieves the state-of-the-art of them in a single single system,” says Huisheng Wang, senior staff software engineer of humans and interactions within the Research and Machine Intelligence division at Google, who was not involved within the work. “This revolutionary system has wide-ranging applications, and has the potential to encourage many future works in the sector of computer vision.” 

Li wrote the paper together with Dina Katabi, the Thuan and Nicole Pham Professor within the MIT Department of Electrical Engineering and Computer Science and a CSAIL principal investigator; Huiwen Chang, a senior research scientist at Google; Shlok Kumar Mishra, a University of Maryland PhD student and Google Research intern; Han Zhang, a senior research scientist at Google; and Dilip Krishnan, a staff research scientist at Google. Computational resources were provided by Google Cloud Platform and the MIT-IBM Watson Research Collaboration. The team’s research was presented on the 2023 Conference on Computer Vision and Pattern Recognition.