Researchers from Intel Labs, in collaboration with academic and industry experts, have introduced a groundbreaking technique for generating realistic and directable human motion from sparse, multi-modal inputs. Their work, highlighted on the European Conference on Computer Vision (ECCV 2024), focuses on overcoming the challenges of generating natural, physically-based human behaviors in high-dimensional humanoid characters. This research is an element of Intel Labs’ broader initiative to advance computer vision and machine learning.
Intel Labs and its partners recently presented six cutting-edge papers at ECCV 2024, a premier conference organized by the European Computer Vision Association (ECVA).
The paper Generating Physically Realistic and Directable Human Motions from Multi-Modal Inputs showcased innovations including a novel defense strategy for safeguarding text-to-image models from prompt-based red teaming attacks and the event of a large-scale dataset designed to enhance spatial consistency in these models. Amongst these contributions, the paper highlights Intel’s dedication to advancing generative modeling while prioritizing responsible AI practices.
Generating Realistic Human Motions Using Multi-Modal Inputs
Intel’s Masked Humanoid Controller (MHC) is a breakthrough system designed to generate human-like motion in simulated physics environments. Unlike traditional methods that rely heavily on fully detailed motion capture data, the MHC is built to handle sparse, incomplete, or partial input data from quite a lot of sources. These sources can include VR controllers, which could only track hand or head movements; joystick inputs that give only high-level navigation commands; video tracking, where certain body parts may be occluded; and even abstract instructions derived from text prompts.
The technology’s innovation lies in its ability to interpret and fill within the gaps where data is missing or incomplete. It achieves this through what Intel terms the Catch-up, Mix, and Complete (CCC) capabilities:
- Catch-up: This feature allows the MHC to recuperate and resynchronize its motion when disruptions occur, equivalent to when the system starts in a failed state, like a humanoid character that has fallen. The system can quickly correct its movements and resume natural motion without retraining or manual adjustments.
- Mix: MHC can mix different motion sequences together, equivalent to merging upper body movements from one motion (e.g., waving) with lower body actions from one other (e.g., walking). This flexibility allows for the generation of entirely recent behaviors from existing motion data.
- Complete: When given sparse inputs, equivalent to partial body movement data or vague high-level directives, the MHC can intelligently infer and generate the missing parts of the motion. For instance, if only arm movements are specified, the MHC can autonomously generate corresponding leg motions to take care of physical balance and realism.
The result’s a highly adaptable motion generation system that may create smooth, realistic, and physically accurate movements, even with incomplete or under-specified directives. This makes MHC ideal for applications in gaming, robotics, virtual reality, and any scenario where high-quality human-like motion is required but input data is proscribed.
The Impact of MHC on Generative Motion Models
The Masked Humanoid Controller (MHC) is an element of a broader effort by Intel Labs and its collaborators to responsibly construct generative models, including people who power text-to-image and 3D generation tasks. As discussed at ECCV 2024, this approach has significant implications for industries like robotics, virtual reality, gaming, and simulation, where the generation of realistic human motion is crucial. By incorporating multi-modal inputs and enabling the controller to seamlessly transition between motions, the MHC can handle real-world conditions where sensor data could also be noisy or incomplete.
This work by Intel Labs stands alongside other advanced research presented at ECCV 2024, equivalent to their novel defense for text-to-image models and the event of techniques for improving spatial consistency in image generation. Together, these advancements showcase Intel’s leadership in the sector of computer vision, with a concentrate on developing secure, scalable, and responsible AI technologies.
Conclusion
The Masked Humanoid Controller (MHC), developed by Intel Labs and academic collaborators, represents a critical step forward in the sector of human motion generation. By tackling the complex control problem of generating realistic movements from multi-modal inputs, the MHC paves the best way for brand spanking new applications in VR, gaming, robotics, and simulation. This research, featured at ECCV 2024, demonstrates Intel’s commitment to advancing responsible AI and generative modeling, contributing to safer and more adaptive technologies across various domains.
