A Powerful 8B Vision-Language Model for the community

We’re excited to release Idefics2, a general multimodal model that takes as input arbitrary sequences of texts and pictures, and generates text responses. It may well answer questions on images, describe visual content, create stories grounded in multiple images, extract information from documents, and perform basic arithmetic operations.
Idefics2 improves upon Idefics1: with 8B parameters, an open license (Apache 2.0), and enhanced OCR (Optical Character Recognition) capabilities, Idefics2 is a powerful foundation for the community working on multimodality. Its performance on Visual Query Answering benchmarks is top of its class size, and competes with much larger models corresponding to LLava-Next-34B and MM1-30B-chat.
Idefics2 can be integrated in 🤗 Transformers from the get-go and subsequently is simple to finetune for a lot of multimodal applications. You possibly can check out the models on the Hub without delay!

* w/ im. split: Following the strategy from SPHINX and LLaVa-NeXT, we allow for an optional sub-image splitting in 4.

Training Data

Idefics2 was trained on a combination of openly available datasets for the pretraining: Interleaved webdocuments (Wikipedia,OBELICS), image-caption pairs (Public Multimodal Dataset, LAION-COCO), OCR data (PDFA (en), IDL and Rendered-text, and image-to-code data (WebSight)).
The interactive visualization allows exploring the OBELICS dataset.
Following common practices in the muse model community, we further train the bottom model on task-oriented data. Nonetheless, these data are sometimes in disparate formats, and scattered in various places. Gathering them is a barrier for the community. To deal with that problem, we’re releasing the multimodal instruction fine-tuning dataset we have been cooking: The Cauldron, an open compilation of 50 manually-curated datasets formatted for multi-turn conversations. We instruction fine-tuned Idefics2 on the concatenation of The Cauldron and various text-only instruction fine-tuning datasets.

Improvements over Idefics1

• We manipulate images of their native resolutions (as much as 980 x 980) and native aspect ratios by following the NaViT strategy. That circumvents the necessity to resize images to fixed-size squares because it has been historically done in the pc vision community. Moreover, we follow the strategy from SPHINX and (optionally) allow sub-image splitting and passing images of very large resolution.
• We significantly enhanced OCR abilities by integrating data that requires the model to transcribe text in a picture or a document. We also improved abilities in answering questions on charts, figures, and documents with appropriate training data.
• We departed from the Idefics1’s architecture (gated cross-attentions) and simplified the mixing of visual features into the language backbone. The photographs are fed to the vision encoder followed by a learned Perceiver pooling and an MLP modality projection. That pooled sequence is then concatenated with the text embeddings to acquire an (interleaved) sequence of image(s) and text(s).

All of those improvements together with higher pre-trained backbones yield a major jump in performance over Idefics1 for a model that’s 10x smaller.

Getting Began with Idefics2

Idefics2 is offered on the Hugging Face Hub and supported within the last transformers version. Here’s a code sample to try it out:

import requests
import torch
from PIL import Image

from transformers import AutoProcessor, AutoModelForVision2Seq
from transformers.image_utils import load_image

DEVICE = "cuda:0"


image1 = load_image("https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-Latest-York-Bay.jpg")
image2 = load_image("https://cdn.britannica.com/59/94459-050-DBA42467/Skyline-Chicago.jpg")
image3 = load_image("https://cdn.britannica.com/68/170868-050-8DDE8263/Golden-Gate-Bridge-San-Francisco.jpg")


processor = AutoProcessor.from_pretrained("HuggingFaceM4/idefics2-8b")
model = AutoModelForVision2Seq.from_pretrained(
    "HuggingFaceM4/idefics2-8b",
).to(DEVICE)



messages = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "What do we see in this image?"},
        ]
    },
    {
        "role": "assistant",
        "content": [
            {"type": "text", "text": "In this image, we can see the city of New York, and more specifically the Statue of Liberty."},
        ]
    },
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "And how about this image?"},
        ]
    },
]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=prompt, images=[image1, image2], return_tensors="pt")
inputs = {k: v.to(DEVICE) for k, v in inputs.items()}



generated_ids = model.generate(**inputs, max_new_tokens=500)
generated_texts = processor.batch_decode(generated_ids, skip_special_tokens=True)

print(generated_texts)

We also provide a fine-tuning colab which should come in useful for anyone seeking to improve Idefics2 on specific use cases.

Resources

Should you want to deep dive further, here is the compilation of all resources for Idefics2:

License

The model is built on top of two pre-trained models: Mistral-7B-v0.1 and siglip-so400m-patch14-384. Each of them have been released under Apache-2.0 license.
We release Idefics2 weights under an Apache-2.0 license as well.

Acknowledgments

Thanks to the Google Team and Mistral AI for releasing and making their models available to the open-source AI community!

Special due to Chun Te Lee for the barplot, and Merve Noyan for the review and suggestions on the blogpost 🤗