Artificial neural networks, ubiquitous machine-learning models that may be trained to finish many tasks, are so called because their architecture is inspired by the best way biological neurons process information within the human brain.

About six years ago, scientists discovered a latest form of more powerful neural network model generally known as a transformer. These models can achieve unprecedented performance, corresponding to by generating text from prompts with near-human-like accuracy. A transformer underlies AI systems corresponding to ChatGPT and Bard, for instance. While incredibly effective, transformers are also mysterious: Unlike with other brain-inspired neural network models, it hasn’t been clear easy methods to construct them using biological components.

Now, researchers from MIT, the MIT-IBM Watson AI Lab, and Harvard Medical School have produced a hypothesis which will explain how a transformer may very well be built using biological elements within the brain. They suggest that a biological network composed of neurons and other brain cells called astrocytes could perform the identical core computation as a transformer.

Recent research has shown that astrocytes, non-neuronal cells which are abundant within the brain, communicate with neurons and play a task in some physiological processes, like regulating blood flow. But scientists still lack a transparent understanding of what these cells do computationally.

With the brand new study, published this week in open-access format within the, the researchers explored the role astrocytes play within the brain from a computational perspective, and crafted a mathematical model that shows how they may very well be used, together with neurons, to construct a biologically plausible transformer.

Their hypothesis provides insights that would spark future neuroscience research into how the human brain works. At the identical time, it could help machine-learning researchers explain why transformers are so successful across a various set of complex tasks.

“The brain is much superior to even the perfect artificial neural networks that we have now developed, but we don’t really know exactly how the brain works. There’s scientific value in occupied with connections between biological hardware and large-scale artificial intelligence networks. That is neuroscience for AI and AI for neuroscience,” says Dmitry Krotov, a research staff member on the MIT-IBM Watson AI Lab and senior writer of the research paper.

Joining Krotov on the paper are lead writer Leo Kozachkov, a postdoc within the MIT Department of Brain and Cognitive Sciences; and Ksenia V. Kastanenka, an assistant professor of neurobiology at Harvard Medical School and an assistant investigator on the Massachusetts General Research Institute.  

A biological impossibility becomes plausible

Transformers operate in a different way than other neural network models. As an example, a recurrent neural network trained for natural language processing would compare each word in a sentence to an internal state determined by the previous words. A transformer, alternatively, compares all of the words within the sentence directly to generate a prediction, a process called self-attention.

For self-attention to work, the transformer must keep all of the words ready in some type of memory, Krotov explains, but this didn’t seem biologically possible as a consequence of the best way neurons communicate.

Nevertheless, a couple of years ago scientists studying a rather different form of machine-learning model (generally known as a Dense Associated Memory) realized that this self-attention mechanism could occur within the brain, but provided that there have been communication between no less than three neurons.

“The number three really popped out to me since it is thought in neuroscience that these cells called astrocytes, which will not be neurons, form three-way connections with neurons, what are called tripartite synapses,” Kozachkov says.

When two neurons communicate, a presynaptic neuron sends chemicals called neurotransmitters across the synapse that connects it to a postsynaptic neuron. Sometimes, an astrocyte can also be connected — it wraps an extended, thin tentacle across the synapse, making a tripartite (three-part) synapse. One astrocyte may form hundreds of thousands of tripartite synapses.

The astrocyte collects some neurotransmitters that flow through the synaptic junction. In some unspecified time in the future, the astrocyte can signal back to the neurons. Because astrocytes operate on a for much longer time scale than neurons — they create signals by slowly elevating their calcium response after which decreasing it — these cells can hold and integrate information communicated to them from neurons. In this fashion, astrocytes can form a form of memory buffer, Krotov says.

“If you happen to give it some thought from that perspective, then astrocytes are extremely natural for precisely the computation we want to perform the eye operation inside transformers,” he adds.

Constructing a neuron-astrocyte network

With this insight, the researchers formed their hypothesis that astrocytes could play a task in how transformers compute. Then they got down to construct a mathematical model of a neuron-astrocyte network that will operate like a transformer.

They took the core mathematics that comprise a transformer and developed easy biophysical models of what astrocytes and neurons do when they convey within the brain, based on a deep dive into the literature and guidance from neuroscientist collaborators.

Then they combined the models in certain ways until they arrived at an equation of a neuron-astrocyte network that describes a transformer’s self-attention.

“Sometimes, we found that certain things we desired to be true couldn’t be plausibly implemented. So, we had to consider workarounds. There are some things within the paper which are very careful approximations of the transformer architecture to give you the chance to match it in a biologically plausible way,” Kozachkov says.

Through their evaluation, the researchers showed that their biophysical neuron-astrocyte network theoretically matches a transformer. As well as, they conducted numerical simulations by feeding images and paragraphs of text to transformer models and comparing the responses to those of their simulated neuron-astrocyte network. Each responded to the prompts in similar ways, confirming their theoretical model.

“Having remained electrically silent for over a century of brain recordings, astrocytes are one of the vital abundant, yet less explored, cells within the brain. The potential of unleashing the computational power of the opposite half of our brain is gigantic,” says Konstantinos Michmizos, associate professor of computer science at Rutgers University, who was not involved with this work. “This study opens up an enchanting iterative loop, from understanding how intelligent behavior may truly emerge within the brain, to translating disruptive hypotheses into latest tools that exhibit human-like intelligence.”

The following step for the researchers is to make the leap from theory to practice. They hope to match the model’s predictions to those which were observed in biological experiments, and use this data to refine, or possibly disprove, their hypothesis.

As well as, one implication of their study is that astrocytes could also be involved in long-term memory, because the network must store information to give you the chance act on it in the longer term. Additional research could investigate this concept further, Krotov says.

“For numerous reasons, astrocytes are extremely necessary for cognition and behavior, they usually operate in fundamentally alternative ways from neurons. My biggest hope for this paper is that it catalyzes a bunch of research in computational neuroscience toward glial cells, and specifically, astrocytes,” adds Kozachkov.

This research was supported, partly, by the BrightFocus Foundation and the National Institute of Health.