Sketch a doodle of a drum or a saxophone to conjure a multi-instrumental composition. Look right into a webcam, speak, and watch your mouth go bouncing across the screen — the input for a series of charmingly clunky chain reactions.

That is what visitors to the MIT Lewis Music Library encounter once they interact with two latest digital installations, “Doodle Tunes” and “Sounds from the Mouth,” created by 2022-23 Center for Art and Technology (CAST) Visiting Artist Andreas Refsgaard in collaboration with Music Technology and Digital Media Librarian Caleb Hall. The residency was initiated by Avery Boddie, Lewis Music Library department head, who recognized Refsgaard’s flair for revealing the playfulness of emerging technologies. The intricacies of coding and machine learning can seem daunting to newcomers, but Refsgaard’s practice as a creative coder, interaction designer, and educator seeks to open the sector to all. Encompassing workshops, an artist talk, class visits, and an exhibition, the residency was infused together with his unique humorousness — a mixture of energetic eccentricity and easygoing relatability.

Learning through laughter

Refsgaard, who relies in Copenhagen, is a real maverick of machine learning. “I’m inquisitive about the ways we are able to express ourselves through code,” he explains. “I wish to make unconventional connections between inputs and outputs, with the pc serving as a translator — a tool might mean you can play music together with your eyes, or it would generate a love poem from a photograph of a burrito.” Refsgaard’s particular spin on innovation isn’t about directly solving problems or launching world-changing startups. As a substitute, he simply seeks to “poke at what may be done,” providing accessible open-source templates to prompt latest creative ideas and applications.

Programmed by Refsgaard and featuring a custom set of sounds created by Hall, “Doodle Tunes” and “Sounds from the Mouth” show how original compositions may be generated through a combination of spontaneous human gestures and algorithmically produced outputs. In “Doodle Tunes,” a machine learning algorithm is trained on a dataset of drawings of various instruments: a piano, drums, bass guitar, or saxophone. When the user sketches certainly one of these images on a touchscreen, a sound is generated; the more instruments you add, the more complex the composition. “Sounds from the Mouth” works through facial tracking and self-capturing images. When the participant faces a webcam and opens their mouth, an autonomous snapshot is created which bounces off the notes of a piano. To try the projects for yourself, scroll to the tip of this text.

Libraries, unlimited

Saxophone squeals and digital drum beats aren’t the one sounds issuing from the areas where the projects are installed. “My office is close by,” says Hall. “So after I suddenly hear laughter, I do know exactly what’s up.” This latest sonic dimension of the Lewis Music Library suits with the ethos of the environment as an entire — designed as a campus hub for audio experimentation, the library was never intended to be wholly silent. Refsgaard’s residency exemplifies a latest emphasis on progressive programming spearheaded by Boddie, because the strategy of the library shifts toward a give attention to digital collections and music technology.

“Along with serving as an area for quiet study and access to physical resources, we would like the library to be a spot where users congregate, collaborate, and explore together,” says Boddie. “This residency was very successful in that regard. Through the workshops, we were capable of connect individuals from across the MIT community and their unique disciplines. We had people from the Sloan School of Management, from the Schwarzman College of Computing, from Music and Theater Arts, all working together, getting messy, creating tools that sometimes worked … and sometimes didn’t.”

Error and serendipity

The mixing of error is a key quality of Refgaard’s work. Occasional glitches are a part of the artistry, and additionally they serve to softly undermine the hype around AI; an algorithm is barely nearly as good as its dataset, and that set is inflected by human biases and oversights. During a public artist talk, “Machine Learning and the Arts,” audience members were initiated into Refsgaard’s offbeat artistic paradigm, presented with projects corresponding to Booksby.ai (an internet bookstore for AI-produced sci-fi novels), Is it FUNKY? (an attempt to differentiate between “fun” and “boring” images), and Eye Conductor (an interface to play music via eye movements and facial gestures). Glitches within the exhibit installations were frankly admitted (it’s true that “Doodle Tunes” occasionally mistakes a drawing of a saxophone for a squirrel), and Refsgaard encouraged audience members to suggest potential improvements.

This open-minded attitude set the tone of the workshops “Art, Algorithms and Artificial Intelligence” and “Machine Learning for Interaction Designers,” intended to be suitable for newcomers in addition to curious experts. Refsgaard’s visits to music technology classes explored the ways in which human creativity may very well be amplified by machine learning, and navigate the sliding scale between artistic intention and unexpected outcomes. “As I see it, success is when participants engage with the fabric and provide you with latest ideas. Step one of learning is to know what’s being taught — the subsequent is to use that understanding in ways in which the teacher couldn’t have foreseen.”

Uncertainty and opportunity

Refsgaard’s work exemplifies a number of the core values and questions central to the evolution of MIT Libraries — problems with digitization, computation, and open access. By selecting to make his lighthearted demos freely accessible, he renounces ownership of his ideas; a machine learning model might function a learning device for a student, and it would equally be monetized by an organization. For Refsgaard, play is a way of engaging with the moral implications of emerging technologies, and Hall found himself grappling with these questions within the technique of creating the sounds for the 2 installations. “If I wrote the sound samples, but another person arranged them as a composition, then who owns the music? Or does the AI own the music? It’s an incredibly interesting time to be working in music technology; we’re stepping into unknown territory.”

For Refsgaard, uncertainty is the key sauce of his algorithmic artistry. “I wish to make things where I’m surprised by the final result,” he says. “I’m searching for that sweet spot between something familiar and something unexpected.” As he explains, an excessive amount of surprise simply amounts to noise, but there’s something joyful in the likelihood that a machine might mistake a saxophone for a squirrel. The duty of a creative coder is to repeatedly tune the connection between human and machine capabilities — to seek out and follow the music.

“Doodle Tunes” and “Sounds from the Mouth” are on display within the MIT Lewis Music Library (14E-109) until Dec. 20. Click the links to interact with the projects online.