Introducing D4RT, a unified AI model for 4D scene reconstruction and tracking across space and time.
Anytime we take a look at the world, we perform a rare feat of memory and prediction. We see and understand things as they’re at a given moment in time, as they were a moment ago, and the way they’re going to be within the moment to follow. Our mental model of the world maintains a persistent representation of reality and we use that model to attract intuitive conclusions concerning the causal relationship between the past, present and future.
To assist machines see the world more like we do, we will equip them with cameras, but that only solves the issue of input. To make sense of this input, computers must solve a fancy, inverse problem: taking a video — which is a sequence of flat 2D projections — and recovering or understanding the wealthy, volumetric 3D world, in motion.
Today, we’re introducing D4RT (Dynamic 4D Reconstruction and Tracking), a brand new AI model that unifies dynamic scene reconstruction right into a single, efficient framework, bringing us closer to the following frontier of artificial intelligence: total perception of our dynamic reality.
The Challenge of the Fourth Dimension
To ensure that it to know a dynamic scene captured on a 2D video, an AI model must track every pixel of each object because it moves through the three dimensions of space and the fourth dimension of time. As well as, it must disentangle this motion from the motion of the camera, maintaining a coherent representation even when objects move behind each other or leave the frame entirely. Traditionally, capturing this level of geometry and motion from 2D videos requires computationally intensive processes or a patchwork of specialised AI models — some for depth, others for movement or camera angles — leading to AI reconstructions which can be slow and fragmented.
D4RT’s simplified architecture and novel query mechanism place it on the forefront of 4D reconstruction while being as much as 300x more efficient than previous methods — fast enough for real-time applications in robotics, augmented reality, and more.
How D4RT Works: A Query-Based Approach
D4RT operates as a unified encoder-decoder Transformer architecture. The encoder first processes the input video right into a compressed representation of the scene’s geometry and motion. Unlike older systems that employed separate modules for various tasks, D4RT calculates only what it needs using a versatile querying mechanism centered around a single, fundamental query:
“Where is a given pixel from the video positioned in 3D space at an arbitrary time, as viewed from a chosen camera?”
Constructing on our prior work, a light-weight decoder then queries this representation to reply specific instances of the posed query. Because queries are independent, they may be processed in parallel on modern AI hardware. This makes D4RT extremely fast and scalable, whether it’s tracking just a couple of points or reconstructing a complete scene.
