For the Image Encoder, they varied between CLIP and AIM models, Image resolution size, and the dataset the models were trained on. The below chart shows you the outcomes for every ablation.

Let’s undergo the foremost pieces above and explain what they’re.

CLIP stands for Contrastive Language Image Pre-training and is supposed to assist your model learn visual concepts by providing names to the things that are supposed to be seen as text. Because the image below shows, this pairs images with text encodings in order that the model will eventually connect the vision tokens (represented within the below image as I, with the text tokens T). This method is named contrastive training.

AIM stands for Autoregressive Image Model, and it’s trained via a reconstructive loss optimization algorithm. The goal here is to see if the transformer can recreate (reconstruct) the image that it’s given.

Image Resolution here refers back to the variety of pixels that’s fed into the transformer. For instance, a 378 x 378 image resolution means we are going to pass in a matrix of that size after which convert it into embeddings that the model will then be trained on. Training Data was split between the (DFN-2B), (DFN-5B), (DFN-5B + VeCap) and (ImageText-400M).

The authors found that image resolution was of highest importance, followed by model size after which the training data contents. Specifically, they saw that the higher the image resolution, the higher the model tended to perform for each zero-shot and few-shot prompting. As more compute is required to coach and run models with higher image resolution requirements, this means that for Vision Transformers, compute will remain of paramount importance.

For the VL Connector, they tested using 64 or 144 tokens for the image, tested using 224, 336, and 378 for the image resolution, and selected between a couple of architectures. I’ll briefly go over the architectures below.

Average Pooling is precisely what it seems like, taking the common of the entire tokens, after which doing a linear projection of this average in order that the grid was 8×8 or 12×12.

Attention Pooling makes the idea that image tokens ought to be treated as samples from a fundamentally different population set than the text tokens. Here we adjust what number of tokens are fed in for every image, within the paper known as k learnable queries. The researchers only considered k of either 64 or 144.

Convolutional Mapping is a a technique from Honeybee that uses a ResNet to dynamically determine what number of tokens to go through to the LLM from the image. That is actualized within the C-Abstractor module.

As you’ll be able to see from the above, different architectures actually had little or no impact. As one might guess, the upper resolution images and the more tokens passed through increased performance amongst the entire connectors but not dramatically so.

This finding suggests we either haven’t found a significantly higher solution to connect the image encoder to the LLM, or that this area is just not where great models will differentiate themselves.

Here, the authors played with 4 different kinds of information: captioned images, synthetically captioned images, interleaved image-text data, and text-only data. They found 4 lessons, each with a graph to summarize the performance changes.

First, interleaving data helps with few-shot and text-only performance, while captioned data helps with zero-shot performance. The researchers varied how much interleaving they did, with the graph below showing the outcomes. As you’ll be able to see, few-shot prompts performed noticeably higher on models trained with interleaved data than the models trained with all or nothing.

Second, Text-only data helps with few-shot reasoning. Text-only on this context signifies that the training data includes image examples and text-only examples. This was done to make sure that the model understands human language in addition to images. Comparing the caption-only to caption-with-text shows a marked improvement for all however the 0-shot reasoning, nevertheless, interleaved-only performs higher than interleaved-plus-text for all however the TextCore test.

Third, when you get the mixture right between image and text you’ll be able to get really strong performance. The above graph shows different ratios of interleaved + captioned data to text-only data. Because the goal is to have a multi-modal model, they never tested the performance when you wouldn’t have any image data. The authors here indicate that the 91/9 ratio produced essentially the most consistently good results.

Fourth, synthetic data helps with few-shot learning. VeCap stands for Visual-enriched Caption, which is a way of making captions so that they’re sure to explain key visual pieces of the image. For the reverse, imagine a caption which will explain the meaning behind a photograph but doesn’t explain any of the weather within the photo. You’ll typically do that in case your data-scraper found images with poor alt-text data.

The authors here concluded that VeCap gives a “non-trivial” boost in few-shot reasoning, but has a comparatively small increase in quality. This raises questions on the cost-effectiveness of VeCap.

Using the outcomes from their ablations, the authors created a Transformer in two-forms: Mixture-of-Expert and regular. Each models had an encoder with a 378 x 378 image, pre-trained with DFN-5B dataset only. That they had a combination of 45% captioned data, 45% interleaved data, and 10% text-only data (approximating the 91:9 ratio of image to text data). The VL Connector had 144 tokens they usually selected a C Abstractor, though they indicate that this was a somewhat arbitrary alternative. For the LLM itself, they created a 3B, 7B, and 30B parameter model (with the MoE model only going as much as 7B). The graph below shows how the these models performed.

Interestingly, the 30B parameter model performs on par with other models which have billions more parameters than it (LLaVA-NeXT-34B, etc.), suggesting that there could also be some quantum relationship between parameter size and performance here.

Multi-modal LLMs are an incredibly exciting a part of the sector. As we discover higher ways to transmit different data types into tokens, we may unlock even greater applications for these transformers. As we glance to the long run, it shouldn’t be unreasonable now to think about how other senses could possibly be inputed outside of a text description, equivalent to sound, smell, and even touch. Data quality is prone to only develop into more priceless.

Because the authors concluded that different language connectors don’t make a significant difference, it’s going to be interesting to see if this implies research should concentrate on the image encoder, or reasonably if we simply haven’t found a real breakthrough solution to use the VL connector.

Outside of this specific paper, one among the large questions that arises is how these MLLMs will perform outside of benchmarks. As LLMs have proliferated, one common criticism revolves around the usage of benchmarks to match them. Often times these benchmarks use a consistent dataset to match, allowing one model to do higher just by overfitting, even when unintentionally. Using methodologies like ELO, the chess rating algorithm, within the LLM Arena from lmsys may give a greater true comparison of model performance.

In closing, as more inputs are capable of be connected to LLMs, one can expect that the variety of applications they may be applied to will increase. Only time will tell how useful we will make this technology.