Stability AI and its multimodal AI research lab, DeepFloyd, have announced the research release of DeepFloyd IF, a cutting-edge text-to-image cascaded pixel diffusion model. The model is initially released under a non-commercial, research-permissible license, but an open-source release is planned for the long run.

DeepFloyd IF boasts several remarkable features, including:

1. Deep text prompt understanding: The model uses T5-XXL-1.1 as a text encoder, with quite a few text-image cross-attention layers, ensuring higher alignment between prompts and pictures.
2. Coherent and clear text alongside generated images: DeepFloyd IF can generate images containing objects with various properties and spatial relations.
3. High degree of photorealism: The model has achieved a formidable zero-shot FID rating of 6.66 on the COCO dataset.
4. Aspect ratio shift: The model can generate images with non-standard aspect ratios, including vertical, horizontal, and the usual square aspect.
5. Zero-shot image-to-image translations: The model can modify a picture’s style, patterns, and details while preserving its basic form.

Below are among the example concepts created by DeepFloyd IF:

DeepFloyd IF’s modular, cascaded, pixel diffusion design consists of several neural modules interacting synergistically. The model works in pixel space, processing high-resolution data in a cascading manner using individually trained models at different resolutions. This involves a base model that generates low-resolution samples and successive super-resolution models that produce high-resolution images.

The model was trained on a custom high-quality LAION-A dataset containing 1 billion (image, text) pairs, a subset of the English a part of the LAION-5B dataset. DeepFloyd’s custom filters were used to remove watermarked, NSFW, and other inappropriate content.

DeepFloyd IF’s process

Initially, DeepFloyd IF is released under a research license. The researchers aim to encourage the event of novel applications across domains corresponding to art, design, storytelling, virtual reality, and accessibility. To encourage potential research, they’ve proposed several technical, academic, and ethical research questions.

Technical research questions include:

• Optimizing the IF model to boost performance, scalability, and efficiency.
• Improving output quality by refining sampling, guiding, or fine-tuning the model.
• Applying techniques used to change Stable Diffusion output to DeepFloyd IF.

Academic research questions include:

• Exploring the role of pre-training for transfer learning.
• Enhancing the model’s control over image generation.
• Expanding the model’s capabilities beyond text-to-image synthesis by integrating multiple modalities.
• Assessing the model’s interpretability to enhance understanding of generated images’ visual features.

Ethical research questions include:

• Identifying and mitigating biases in DeepFloyd IF.
• Assessing the model’s impact on social media and content generation.
• Developing an efficient fake image detector that utilizes the model.

To access the model’s weights, users must accept the license on DeepFloyd’s Hugging Face space. For more information, you’ll be able to visit the model’s website, GitHub repository, Gradio demo, or join public discussions through DeepFloyd’s Linktree.