As we experience the world, our senses (vision, sounds, smells) provide a various array of data, and we express ourselves using different communication methods, similar to facial expressions and gestures. These senses and communication methods are collectively called modalities, representing the various ways we perceive and communicate. Drawing inspiration from this human capability, large multimodal model (LMM), a mixture of generative and multimodal AI, are being developed to know and create content using differing types like text, images, and audio. In this text, we delve into this newly emerging field, exploring what LMMs (Large Multimodal Models) are, how they’re constructed, existing examples, the challenges they face, and potential applications.

Evolution of Generative AI in 2024: From Large Language Models to Large Multimodal Models

In its latest report, McKinsey designated 2023 as a breakout 12 months for generative AI, resulting in many advancements in the sector. Now we have witnessed a notable rise within the prevalence of enormous language models (LLMs) adept at understanding and generating human-like language. Moreover, image generation models are significantly evolved, demonstrating their ability to create visuals from textual prompts. Nonetheless, despite significant progress in individual modalities like text, images, or audio, generative AI has encountered challenges in seamlessly combining these modalities within the generation process. Because the world is inherently multimodal in nature, it’s crucial for AI to grapple with multimodal information. This is crucial for meaningful engagement with humans and successful operation in real-world scenarios.

Consequently, many AI researchers anticipate the rise of LMMs as the following frontier in AI research and development in 2024. This evolving frontier focuses on enhancing the capability of generative AI to process and produce diverse outputs, spanning text, images, audio, video, and other modalities. It is crucial to emphasise that not all multimodal systems qualify as LMMs. Models like Midjourney and Stable Diffusion, despite being multimodal, don’t fit into the LMM category mainly because they lack the presence of LLMs, that are a fundamental component of LMMs. In other words, we are able to describe LMMs as an extension of LLMs, providing them with the potential to proficiently handle various modalities.

How do LMMs Work?

While researchers have explored various approaches to constructing LMMs, they typically involve three essential components and operations. First, encoders are employed for every data modality to generate data representations (known as embeddings) specific to that modality. Second, different mechanisms are used for aligning embeddings from different modalities right into a unified multimodal embedding space. Third, for generative models, an LLM is employed to generate text responses. As inputs may consist of text, images, videos and audios, researchers are working on latest ways to make language models consider different modalities when giving responses.

Development of LMMs in 2023

Below, I actually have briefly outlined a few of the notable LMMs developed in 2023.

• LLaVA is an open-source LMM, jointly developed by the University of Wisconsin-Madison, Microsoft Research, and Columbia University. The model goals to supply an open-source version of multimodal GPT4. Leveraging Meta’s Llama LLM, it incorporates the CLIP visual encoder for robust visual comprehension. The healthcare-focused variant of LLaVa, termed as LLaVA-Med, can answer inquiries related to biomedical images.
• ImageBind is an open-source model crafted by Meta, emulating the power of human perception to relate multimodal data. The model integrates six modalities—text, images/videos, audio, 3D measurements, temperature data, and motion data—learning a unified representation across these diverse data types. ImageBind can connect objects in photos with attributes like sound, 3D shapes, temperature, and motion. The model will be used, as an example, to generate scene from text or sounds.
• SeamlessM4T is a multimodal model designed by Meta to foster communication amongst multilingual communities. SeamlessM4T excels in translation and transcription tasks, supporting speech-to-speech, speech-to-text, text-to-speech, and text-to-text translations. The model employs non-autoregressive text-to-unit decoder to perform these translations. The improved version, SeamlessM4T v2, forms the idea for models like SeamlessExpressive and SeamlessStreaming, emphasizing the preservation of expression across languages and delivering translations with minimal latency.
• GPT4, launched by OpenAI, is an advancement of its predecessor, GPT3.5. Although detailed architectural specifics aren’t fully disclosed, GPT4 is well-regarded for its smooth integration of text-only, vision-only, and audio-only models. The model can generate text from each written and graphical inputs. It excels in various tasks, including humor description in images, summarization of text from screenshots, and responding adeptly to exam questions featuring diagrams. GPT4 can be recognized for its adaptability in effectively processing a big selection of input data formats.
• Gemini, created by Google DeepMind, distinguishes itself by being inherently multimodal, allowing seamless interaction across various tasks without counting on stitching together single-modality components. This model effortlessly manages each text and diverse audio-visual inputs, showcasing its capability to generate outputs in each text and image formats.

Challenges of Large Multimodal Models

• Incorporating More Data Modalities: Most of existing LMMs operate with text and pictures. Nonetheless, LMMs have to evolve beyond text and pictures, accommodating modalities like videos, music, and 3D.
• Diverse Dataset Availability: Considered one of the important thing challenges in developing and training multimodal generative AI models is the necessity for giant and diverse datasets that include multiple modalities. For instance, to coach a model to generate text and pictures together, the dataset needs to incorporate each text and image inputs which can be related to one another.
• Generating Multimodal Outputs: While LMMs can handle multimodal inputs, generating diverse outputs, similar to combining text with graphics or animations, stays a challenge.
• Following Instructions: LMMs face the challenge of mastering dialogue and instruction-following tasks, moving beyond mere completion.
• Multimodal Reasoning: While current LMMs excel at transforming one modality into one other, the seamless integration of multimodal data for complex reasoning tasks, like solving written word problems based on auditory instructions, stays a difficult endeavor.
• Compressing LMMs: The resource-intensive nature of LMMs poses a big obstacle, rendering them impractical for edge devices with limited computational resources. Compressing LMMs to reinforce efficiency and make them suitable for deployment on resource-constrained devices is a vital area of ongoing research.

Potential Use Cases

• Education: LMMs have the potential to remodel education by generating diverse and interesting learning materials that mix text, images, and audio. LMMs provide comprehensive feedback on assignments, promote collaborative learning platforms, and enhance skill development through interactive simulations and real-world examples.
• Healthcare: In contrast to traditional AI diagnostic systems that focus on a single modality, LMMs improve medical diagnostics by integrating multiple modalities. In addition they support communication across language barriers amongst healthcare providers and patients, acting as a centralized repository for various AI applications inside hospitals.
• Art and Music Generation: LMMs could excel in art and music creation by combining different modalities for unique and expressive outputs. For instance, an art LMM can mix visual and auditory elements, providing an immersive experience. Likewise, a music LMM can integrate instrumental and vocal elements, leading to dynamic and expressive compositions.
• Personalized Recommendations: LMMs can analyze user preferences across various modalities to supply personalized recommendations for content consumption, similar to movies, music, articles, or products.

• Weather Prediction and Environmental Monitoring: LMMs can analyze various modalities of knowledge, similar to satellite images, atmospheric conditions, and historical patterns, to enhance accuracy in weather prediction and environmental monitoring.

The Bottom Line

The landscape of Large Multimodal Models (LMMs) marks a big breakthrough in generative AI, promising advancements in various fields. As these models seamlessly integrate different modalities, similar to text, images, and audio, their development opens doors to transformative applications in healthcare, education, art, and personalized recommendations. Nonetheless, challenges, including accommodating more data modalities and compressing resource-intensive models, underscore the continuing research efforts needed for the complete realization of LMMs’ potential.