M.C. Escher’s artwork is a gateway right into a world of depth-defying optical illusions, featuring “unimaginable objects” that break the laws of physics with convoluted geometries. What you perceive his illustrations to be will depend on your viewpoint — for instance, an individual seemingly walking upstairs could also be heading down the steps for those who tilt your head sideways.
Computer graphics scientists and designers can recreate these illusions in 3D, but only by bending or cutting an actual shape and positioning it at a selected angle. This workaround has downsides, though: Changing the smoothness or lighting of the structure will expose that it isn’t actually an optical illusion, which also means you possibly can’t accurately solve geometry problems on it.
Researchers at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) have developed a singular approach to represent “unimaginable” objects in a more versatile way. Their “Meschers” tool converts images and 3D models into 2.5-dimensional structures, creating Escher-like depictions of things like windows, buildings, and even donuts. The approach helps users relight, smooth out, and study unique geometries while preserving their optical illusion.
This tool could assist geometry researchers with calculating the gap between two points on a curved unimaginable surface (“geodesics”) and simulating how heat dissipates over it (“heat diffusion”). It could also help artists and computer graphics scientists create physics-breaking designs in multiple dimensions.
Lead creator and MIT PhD student Ana Dodik goals to design computer graphics tools that aren’t limited to replicating reality, enabling artists to precise their intent independently of whether a shape will be realized within the physical world. “Using Meschers, we’ve unlocked a brand new class of shapes for artists to work with on the pc,” she says. “They might also help perception scientists understand the purpose at which an object truly becomes unimaginable.”
Dodik and her colleagues will present their paper on the SIGGRAPH conference in August.
Making unimaginable objects possible
Inconceivable objects can’t be fully replicated in 3D. Their constituent parts often look plausible, but these parts don’t glue together properly when assembled in 3D. But what will be computationally imitated, because the CSAIL researchers came upon, is the means of how we perceive these shapes.
Take the Penrose Triangle, as an example. The item as a complete is physically unimaginable since the depths don’t “add up,” but we will recognize real-world 3D shapes (like its three L-shaped corners) inside it. These smaller regions will be realized in 3D — a property called “local consistency” — but after we attempt to assemble them together, they don’t form a globally consistent shape.
The Meschers approach models’ locally consistent regions without forcing them to be globally consistent, piecing together an Escher-esque structure. Behind the scenes, Meschers represents unimaginable objects as if we all know their x and y coordinates within the image, in addition to differences in z coordinates (depth) between neighboring pixels; the tool uses these differences in depth to reason about unimaginable objects not directly.
The numerous uses of Meschers
Along with rendering unimaginable objects, Meschers can subdivide their structures into smaller shapes for more precise geometry calculations and smoothing operations. This process enabled the researchers to cut back visual imperfections of unimaginable shapes, similar to a red heart outline they thinned out.
The researchers also tested their tool on an “impossibagel,” where a bagel is shaded in a physically unimaginable way. Meschers helped Dodik and her colleagues simulate heat diffusion and calculate geodesic distances between different points of the model.
“Imagine you’re an ant traversing this bagel, and you wish to understand how long it’ll take you to get across, for instance,” says Dodik. “In the identical way, our tool could help mathematicians analyze the underlying geometry of unimaginable shapes up close, very like how we study real-world ones.”
Very similar to a magician, the tool can create optical illusions out of otherwise practical objects, making it easier for computer graphics artists to create unimaginable objects. It might also use “inverse rendering” tools to convert drawings and pictures of unimaginable objects into high-dimensional designs.
“Meschers demonstrates how computer graphics tools don’t need to be constrained by the principles of physical reality,” says senior creator Justin Solomon, associate professor of electrical engineering and computer science and leader of the CSAIL Geometric Data Processing Group. “Incredibly, artists using Meschers can reason about shapes that we’ll never find in the true world.”
Meschers may aid computer graphics artists with tweaking the shading of their creations, while still preserving an optical illusion. This versatility would allow creatives to alter the lighting of their art to depict a greater variety of scenes (like a sunrise or sunset) — as Meschers demonstrated by relighting a model of a dog on a skateboard.
Despite its versatility, Meschers is just the beginning for Dodik and her colleagues. The team is considering designing an interface to make the tool easier to make use of while constructing more elaborate scenes. They’re also working with perception scientists to see how the pc graphics tool will be used more broadly.
Dodik and Solomon wrote the paper with CSAIL affiliates Isabella Yu ’24, SM ’25; PhD student Kartik Chandra SM ’23; MIT professors Jonathan Ragan-Kelley and Joshua Tenenbaum; and MIT Assistant Professor Vincent Sitzmann.
Their work was supported, partly, by the MIT Presidential Fellowship, the Mathworks Fellowship, the Hertz Foundation, the U.S. National Science Foundation, the Schmidt Sciences AI2050 fellowship, MIT Quest for Intelligence, the U.S. Army Research Office, U.S. Air Force Office of Scientific Research, SystemsThatLearn@CSAIL initiative, Google, the MIT–IBM Watson AI Laboratory, from the Toyota–CSAIL Joint Research Center, Adobe Systems, the Singapore Defence Science and Technology Agency, and the U.S. Intelligence Advanced Research Projects Activity.
