Benjamin Warf, a renowned neurosurgeon at Boston Children’s Hospital, stands within the MIT.nano Immersion Lab. Greater than 3,000 miles away, his virtual avatar stands next to Matheus Vasconcelos in Brazil because the resident practices delicate surgery on a doll-like model of a baby’s brain.

With a pair of virtual-reality goggles, Vasconcelos is capable of watch Warf’s avatar display a brain surgery procedure before replicating the technique himself and while asking questions of Warf’s digital twin.

“It’s an almost out-of-body experience,” Warf says of watching his avatar interact with the residents. “Perhaps it’s the way it feels to have an analogous twin?”

And that’s the goal: Warf’s digital twin bridged the gap, allowing him to be functionally in two places without delay. “It was my first training using this model, and it had excellent performance,” says Vasconcelos, a neurosurgery resident at Santa Casa de São Paulo School of Medical Sciences in São Paulo, Brazil. “As a resident, I now feel more confident and comfy applying the technique in an actual patient under the guidance of a professor.”

Warf’s avatar arrived via a recent project launched by medical simulator and augmented reality (AR) company EDUCSIM. The corporate is an element of the 2023 cohort of START.nano, MIT.nano’s deep-tech accelerator that provides early-stage startups discounted access to MIT.nano’s laboratories.

In March 2023, Giselle Coelho, EDUCSIM’s scientific director and a pediatric neurosurgeon at Santa Casa de São Paulo and Sabará Children’s Hospital, began working with technical staff within the MIT.nano Immersion Lab to create Warf’s avatar. By November, the avatar was training future surgeons like Vasconcelos.

“I had this concept to create the avatar of Dr. Warf as a proof of concept, and asked, ‘What could be the place on this planet where they’re working on technologies like that?’” Coelho says. “Then I discovered MIT.nano.”

Capturing a Surgeon

As a neurosurgery resident, Coelho was so frustrated by the dearth of practical training options for complex surgeries that she built her own model of a baby brain. The physical model incorporates all of the structures of the brain and may even bleed, “simulating all of the steps of a surgery, from incision to skin closure,” she says.

She soon found that simulators and virtual reality (VR) demonstrations reduced the training curve for her own residents. Coelho launched EDUCSIM in 2017 to expand the variability and reach of the training for residents and experts seeking to learn recent techniques.

Those techniques include a procedure to treat infant hydrocephalus that was pioneered by Warf, the director of neonatal and congenital neurosurgery at Boston Children’s Hospital. Coelho had learned the technique directly from Warf and thought his avatar is perhaps the way in which for surgeons who couldn’t travel to Boston to profit from his expertise.

To create the avatar, Coelho worked with Talis Reks, the AR/VR/gaming/big data IT technologist within the Immersion Lab.

“Numerous technology and hardware might be very expensive for startups to access as they begin their company journey,” Reks explains. “START.nano is a technique of enabling them to utilize and afford the tools and technologies we’ve at MIT.nano’s Immersion Lab.”

Coelho and her colleagues needed high-fidelity and high-resolution motion-capture technology, volumetric video capture, and a spread of other VR/AR technologies to capture Warf’s dexterous finger motions and facial expressions. Warf visited MIT.nano on several occasions to be digitally “captured,” including performing an operation on the physical baby model while wearing special gloves and clothing embedded with sensors.

“These technologies have mostly been used for entertainment or VFX [visual effects] or CGI [computer-generated imagery],” says Reks, “But this can be a unique project, because we’re applying it now for real medical practice and real learning.”

One among the largest challenges, Reks says, was helping to develop what Coelho calls “holoportation”— transmitting the 3D, volumetric video capture of Warf in real-time over the web in order that his avatar can appear in transcontinental medical training.

The Warf avatar has synchronous and asynchronous modes. The training that Vasconcelos received was within the asynchronous mode, where residents can observe the avatar’s demonstrations and ask it questions. The answers, delivered in a wide range of languages, come from AI algorithms that draw from previous research and an in depth bank of questions and answers provided by Warf.

Within the synchronous mode, Warf operates his avatar from a distance in real time, Coelho says. “He could walk across the room, he could refer to me, he could orient me. It’s amazing.”

Coelho, Warf, Reks, and other team members demonstrated a mixture of the modes in a second session in late December. This demo consisted of volumetric live video capture between the Immersion Lab and Brazil, spatialized and visual in real-time through AR headsets. It significantly expanded upon the previous demo, which had only streamed volumetric data in a single direction through a two-dimensional display.

Powerful impacts

Warf has a protracted history of coaching desperately needed pediatric neurosurgeons all over the world, most recently through his nonprofit Neurokids. Distant and simulated training has been an increasingly large part of coaching because the pandemic, he says, although he doesn’t feel it can ever completely replace personal hands-on instruction and collaboration.

“But when in actual fact sooner or later we could have avatars, like this one from Giselle, in foreign places showing people do things and answering questions for them, without the associated fee of travel, without the time cost and so forth, I believe it might be really powerful,” Warf says.

The avatar project is particularly essential for surgeons serving distant and underserved areas just like the Amazon region of Brazil, Coelho says. “It is a method to give them the identical level of education that they might get in different places, and the identical opportunity to be in contact with Dr. Warf.”

One baby treated for hydrocephalus at a recent Amazon clinic had traveled by boat 30 hours for the surgery, in line with Coelho.

Training surgeons with the avatar, she says, “can change reality for this baby and may change the longer term.”