Imagine a boombox that tracks your every move and suggests music to match your personal dance style. That’s the thought behind “Be the Beat,” certainly one of several projects from MIT course 4.043/4.044 (Interaction Intelligence), taught by Marcelo Coelho within the Department of Architecture, that were presented on the thirty eighth annual NeurIPS (Neural Information Processing Systems) conference in December 2024. With over 16,000 attendees converging in Vancouver, NeurIPS is a competitive and prestigious conference dedicated to research and science in the sphere of artificial intelligence and machine learning, and a premier venue for showcasing cutting-edge developments.
The course investigates the emerging field of large language objects, and the way artificial intelligence will be prolonged into the physical world. While “Be the Beat” transforms the creative possibilities of dance, other student submissions span disciplines akin to music, storytelling, critical pondering, and memory, creating generative experiences and recent types of human-computer interaction. Taken together, these projects illustrate a broader vision for artificial intelligence: one which goes beyond automation to catalyze creativity, reshape education, and reimagine social interactions.
Be the Beat
“Be the Beat,” by Ethan Chang, an MIT mechanical engineering and design student, and Zhixing Chen, an MIT mechanical engineering and music student, is an AI-powered boombox that implies music from a dancer’s movement. Dance has traditionally been guided by music throughout history and across cultures, yet the concept of dancing to create music is never explored.
“Be the Beat” creates an area for human-AI collaboration on freestyle dance, empowering dancers to rethink the normal dynamic between dance and music. It uses PoseNet to explain movements for a big language model, enabling it to investigate dance style and query APIs to search out music with similar style, energy, and tempo. Dancers interacting with the boombox reported having more control over artistic expression and described the boombox as a novel approach to discovering dance genres and choreographing creatively.
A Mystery for You
“A Mystery for You,” by Mrinalini Singha SM ’24, a recent graduate within the Art, Culture, and Technology program, and Haoheng Tang, a recent graduate of the Harvard University Graduate School of Design, is an academic game designed to cultivate critical pondering and fact-checking skills in young learners. The sport leverages a big language model (LLM) and a tangible interface to create an immersive investigative experience. Players act as citizen fact-checkers, responding to AI-generated “news alerts” printed by the sport interface. By inserting cartridge combos to prompt follow-up “news updates,” they navigate ambiguous scenarios, analyze evidence, and weigh conflicting information to make informed decisions.
This human-computer interaction experience challenges our news-consumption habits by eliminating touchscreen interfaces, replacing perpetual scrolling and skim-reading with a haptically wealthy analog device. By combining the affordances of slow media with recent generative media, the sport promotes thoughtful, embodied interactions while equipping players to raised understand and challenge today’s polarized media landscape, where misinformation and manipulative narratives thrive.
Memorscope
“Memorscope,” by MIT Media Lab research collaborator Keunwook Kim, is a tool that creates collective memories by merging the deeply human experience of face-to-face interaction with advanced AI technologies. Inspired by how we use microscopes and telescopes to look at and uncover hidden and invisible details, Memorscope allows two users to “look into” one another’s faces, using this intimate interaction as a gateway to the creation and exploration of their shared memories.
The device leverages AI models akin to OpenAI and Midjourney, introducing different aesthetic and emotional interpretations, which ends up in a dynamic and collective memory space. This space transcends the restrictions of traditional shared albums, offering a fluid, interactive environment where memories are usually not just static snapshots but living, evolving narratives, shaped by the continuing relationship between users.
Narratron
“Narratron,” by Harvard Graduate School of Design students Xiying (Aria) Bao and Yubo Zhao, is an interactive projector that co-creates and co-performs kid’s stories through shadow puppetry using large language models. Users can press the shutter to “capture” protagonists they wish to be within the story, and it takes hand shadows (akin to animal shapes) as input for the important characters. The system then develops the story plot as recent shadow characters are introduced. The story appears through a projector as a backdrop for shadow puppetry while being narrated through a speaker as users turn a crank to “play” in real time. By combining visual, auditory, and bodily interactions in a single system, the project goals to spark creativity in shadow play storytelling and enable multi-modal human-AI collaboration.
Perfect Syntax
“Perfect Syntax,” by Karyn Nakamura ’24, is a video art piece examining the syntactic logic behind motion and video. Using AI to govern video fragments, the project explores how the fluidity of motion and time will be simulated and reconstructed by machines. Drawing inspiration from each philosophical inquiry and artistic practice, Nakamura’s work interrogates the connection between perception, technology, and the movement that shapes our experience of the world. By reimagining video through computational processes, Nakamura investigates the complexities of how machines understand and represent the passage of time and motion.
