An 11B parameter pretrained language model and VLM, trained on over 5000B tokens and 11 languages

The Falcon 2 Models

TII is launching a brand new generation of models, Falcon 2, focused on providing the open-source community with a series of smaller models with enhanced performance and multi-modal support. Our goal is to enable cheaper inference and encourage the event of more downstream applications with improved usability.

The primary generation of Falcon models, featuring Falcon-40B and Falcon-180B, made a major contribution to the open-source community, promoting the discharge of advanced LLMs with permissive licenses. More detailed information on the previous generation of Falcon models could be present in the RefinedWeb, Penedo et al., 2023 and The Falcon Series of Open Language Models, Almazrouei et al., 2023 papers, and the Falcon and Falcon-180B blog posts.

The second generation of models is concentrated on increased usability and integrability, constructing a multi-modal ecosystem. We start this journey by releasing not only the bottom 11B LLM, but in addition the 11B VLM model that comes with image understanding capabilities. The vision-language model, or VLM, will allow users to have interaction in chats about visual content using text.

As with our previous work, the models offer support mainly in English but have good capabilities in ten other languages, including Spanish, French, and German.

Table of Contents

Falcon2-11B LLM

Training Data

Falcon2-11B was trained on over 5,000 GT (billion tokens) of RefinedWeb, a high-quality filtered and deduplicated web dataset, enhanced with curated corpora. It followed a four-stage training strategy. The primary three stages were focused on increasing the context length, from 2048 to 4096 and at last to 8192 tokens. The last stage aimed to further enhance performance using only high-quality data.

Overall, the info sources included RefinedWeb-English, RefinedWeb-Europe (cs, de, es, fr, it, nl, pl, pt, ro, sv), high-quality technical data, code data, and conversational data extracted from public sources.

The training stages were as follows:

The information was tokenized with Falcon2-11B tokenizer, the identical tokenizer as for the previous Falcon models.

Model Architecture

The next table summarizes a number of the crucial details concerning the model architecture:

Training Procedure

Falcon2-11B was trained on 1024 A100 40GB GPUs for nearly all of the training, using a 3D parallelism strategy (TP=8, PP=1, DP=128) combined with ZeRO and Flash-Attention 2.

Training Hyperparameters

Falcon2-11B Evaluation

English performance

Performance on Open LLM Leaderboard tasks:

The Hugging Face Leaderboard team provided an official evaluation of our model on the Open LLM Leaderboard tasks. The model performs higher than models comparable to Llama3-8B (trained on thrice more data) and Mistral-7B, and on par with Gemma-7b.

Zero shot performance:

The evaluation results show that the Falcon2-11B shows similar performance to Falcon-40B, at a 4 times smaller model size!

Multilingual capabilities

Using the Multilingual LLM leaderboard, we compare the Falcon2-11B model to the Llama-7B and Bloom-7B. For reference, we also include Falcon-40B (that supports the identical languages), Falcon-7B (that supports French) and Mistral-7B.

Within the spirit of the unique Falcon models, the Falcon2-11B was trained not only on English data but in addition on ten other languages. Our multilingual evaluation results show that the model presents good capabilities within the six languages (de, es, fr, it, nl, ro) featured on the Multilingual LLM Leaderboard and truly shows higher performance than the Falcon-40B and a number of other other multilingual models on all of the cited languages.

We’ll soon release more extensive evaluation results for multilingual capabilities within the Falcon2-11B model card!

Code generation capabilities

We check the model’s performance on code generation against the BigCode Leaderboard on the HumanEval benchmark for the Python language, obtaining pass@1 of 29.59%.

Using Falcon2-11B

from transformers import AutoTokenizer
import transformers
import torch

model = "tiiuae/falcon-11B"

tokenizer = AutoTokenizer.from_pretrained(model)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

After which, you’d run text generation using code like the next:

sequences = pipeline(
   "Are you able to explain the concept of Quantum Computing?",
    max_length=200,
    do_sample=True,
    top_k=10,
    num_return_sequences=1,
    eos_token_id=tokenizer.eos_token_id,
)
for seq in sequences:
    print(f"Result: {seq['generated_text']}")

Falcon2-11B VLM

Falcon2-11B VLM is a vision-language model (VLM) built on top of the LLM, that moreover handles image inputs and is able to answering queries concerning the images. To attain this, we integrate the pretrained CLIP ViT-L/14 vision encoder with our Falcon2-11B chat-finetuned model, and train with image-text data.

To boost the VLM’s perception of fine-grained details w.r.t small objects in images, we employ a dynamic encoding mechanism at high-resolution for image inputs, much like LLaVA-Next.

Training

The training is finished in two stages: pretraining and finetuning. In each stages, the visual encoder weights are kept frozen. Within the pretraining stage, the LLM is kept frozen, and only the multimodal projector is trained on 558K image-caption pairs.
This permits the multimodal projector to learn a mapping from visual to text embedding space. During finetuning, each the projector and LLM weights are trained on a corpus of 1.2M image-text instruction data from public datasets, which also includes multi-round conversations.

Falcon2-11B VLM Evaluation

Using Falcon2-11B-FalconVLM

from transformers import LlavaNextForConditionalGeneration, LlavaNextProcessor
from PIL import Image
import requests
import torch

processor = LlavaNextProcessor.from_pretrained("tiiuae/falcon-11B-vlm")
model = LlavaNextForConditionalGeneration.from_pretrained("tiiuae/falcon-11B-vlm", torch_dtype=torch.bfloat16)

url = "https://merzougabirding.com/wp-content/uploads/2023/09/falcon-size.jpg"
falcon_image = Image.open(requests.get(url, stream=True).raw)
prompt = "User: nWhat's special about this bird's vision?"

inputs = processor(prompt, images=falcon_image, return_tensors="pt", padding=True).to('cuda:0')

model.to('cuda:0')
output = model.generate(**inputs, max_new_tokens=256)


prompt_length = inputs['input_ids'].shape[1]
generated_captions = processor.decode(output[0], skip_special_tokens=True).strip()

print(generated_captions)

License information

The Falcon 2 models are made available under the TII Falcon 2 License, a permissive Apache 2.0-based software license which incorporates an acceptable use policy that promotes the responsible use of AI. This license was crafted throughout the spirit of TII’s commitment to the open source community.