Neuromorphic Computing — an Edgier, Greener AI

Why computer hardware and AI algorithms are being reinvented using inspiration from the brain

There are periodic proclamations of the approaching neuromorphic computing revolution, which uses inspiration from the brain to rethink neural networks and the hardware they run on. While there remain challenges in the sphere, there have been solid successes and continues to be regular progress in spiking neural network algorithms and neuromorphic hardware. This progress is paving the best way for disruption in a minimum of some sectors of artificial intelligence and can reduce the energy consumption per computation at inference and permit artificial intelligence to be pushed further out to the sting. In this text, I’ll cover some neuromorphic computing and engineering basics, training, some great benefits of neuromorphic systems, and the remaining challenges.

The classical use case of neuromorphic systems is for edge devices that have to perform the computation locally and are energy-limited, for instance, battery-powered devices. Nevertheless, one in every of the recent interests in using neuromorphic systems is to scale back energy usage at data centers, reminiscent of the energy needed by large language models (LLMs). For instance, OpenAI signed a letter of intent to buy $51 million of neuromorphic chips from Rain AI in December 2023. This is smart since OpenAI spends so much on inference, with one estimate of around $4 billion on running inference in 2024. It also appears that each Intel’s Loihi 2 and IBM’s NorthPole (successor to TrueNorth) neuromorphic systems are designed to be used in servers.

The guarantees of neuromorphic computing can broadly be divided into 1) pragmatic, near-term successes which have already found successes and a pair of) more aspirational, wacky neuroscientist fever-dream ideas of how spiking dynamics might endow neural networks with something closer to real intelligence. In fact, it’s group 2 that actually excites me, but I’m going to concentrate on group 1 for this post. And there isn’t a more exciting option to start than to dive into terminology.

Neuromorphic computation is usually defined as computation that’s brain-inspired, but that definition leaves so much to the imagination. Neural networks are more neuromorphic than classical computation, but today neuromorphic computation is specifically curious about using event-based spiking neural networks (SNNs) for his or her energy efficiency. Regardless that SNNs are a variety of artificial neural network, the term “artificial neural networks” (ANNs) is reserved for the more standard non-spiking artificial neural networks within the neuromorphic literature. Schuman and colleagues (2022) define neuromorphic computers as non-von Neuman computers where each processing and memory are collocated in artificial neurons and synapses, versus von Neuman computers that separate processing and memory.

Neuromorphic engineering means designing the hardware while “neuromorphic computation” is targeted on what’s being simulated fairly than what it’s being simulated on. These are tightly intertwined because the computation relies on the properties of the hardware and what’s implemented in hardware is dependent upon what’s empirically found to work best.

One other related term is NeuroAI, the goal of which is to make use of AI to achieve a mechanistic understanding of the brain and is more curious about biological realism. Neuromorphic computation is curious about neuroscience as a method to an end. It views the brain as a source of ideas that might be used to attain objectives reminiscent of energy efficiency and low latency in neural architectures. An honest amount of the NeuroAI research relies on spike averages fairly than spiking neural networks, which allows closer comparison of nearly all of modern ANNs which can be applied to discrete tasks.

Neuromorphic systems are event-based, which is a paradigm shift from how modern ANN systems work. Even real-time ANN systems typically process one frame at a time, with activity synchronously propagated from one layer to the following. Which means that in ANNs, neurons that carry no information require the identical processing as neurons that carry critical information. Event-driven is a unique paradigm that usually starts on the sensor and applies essentially the most work where information must be processed. ANNs depend on matrix operations that take the identical period of time and energy whatever the values within the matrices. Neuromorphic systems use SNNs where the quantity of labor is dependent upon the variety of spikes.

A conventional deployed ANN would often be connected to a camera that synchronously records a frame in a single exposure. The ANN then processes the frame. The outcomes of the frame might then be fed right into a tracking algorithm and further processed.

Event-driven systems may start on the sensor with an event camera. Each pixel sends updates asynchronously every time a change crosses a threshold. So when there may be movement in a scene that’s otherwise stationary, the pixels that correspond to the movement send events or spikes immediately without waiting for a synchronization signal. The event signals might be sent inside tens of microseconds, while a conventional camera might collect at 24 Hz and will introduce a latency that’s within the range of tens of milliseconds. Along with receiving the knowledge sooner, the knowledge within the event-based system can be sparser and would concentrate on the movement. The normal system would need to process all the scene through each network layer successively.

One in all the foremost challenges of SNNs is training them. Backpropagation algorithms and stochastic gradient descent are the go-to solutions for training ANNs, nevertheless, these methods run into difficulty with SNNs. The most effective option to train SNNs is just not yet established and the next methods are a few of the more common approaches which can be used:

1. ANN to SNN conversion
2. Backpropagation-like
3. Synaptic plasticity
4. Evolutionary

ANN to SNN conversion

One method of making SNNs is to bypass training the SNNs directly and as an alternative train ANNs. This approach limits the kinds of SNNs and hardware that might be used. For instance, Sengupta et al. (2019) converted VGG and ResNets to ANNs using an integrate-and-fire (IF) neuron that doesn’t have a leaking or refractory period. They introduce a novel weight-normalization technique to perform the conversion, which involves setting the firing threshold of every neuron based on its pre-synaptic weights. Dr. Priyadarshini Panda goes into more detail in her ESWEEK 2021 SNN Talk.

Benefits:

1. Enables deep SNNs.
2. Allows reuse of deep ANN knowledge, reminiscent of training, architecture, etc.

Disadvantages:

1. Limits architectures to those suited to ANNs and the conversion procedures.
2. Network doesn’t learn to make the most of SNN properties, which may result in lower accuracy and longer latency.

Backpropagation-like approaches and surrogate gradient descent

Probably the most common methods currently used to coach SNNs are backpropagation-like approaches. Standard backpropagation doesn’t work to coach SNNs because 1) the spiking threshold function’s gradient is nonzero except at the edge where it’s undefined and a pair of) the credit task problem must be solved within the temporal dimension as well as spatial (or color etc).

In ANNs, essentially the most common activation function is the ReLU. For SNNs, the neuron will fire if the membrane potential is above some threshold, otherwise, it’ll not fire. This is named a Heaviside function. You may use a sigmoid function as an alternative, but then it will not be a spiking neural network. The answer of using surrogate gradients is to make use of the usual threshold function within the forward pass, but then use the derivative from a “smoothed” version of the Heaviside function, reminiscent of the sigmoid function, within the backward pass (Neftci et al. 2019, Bohte 2011).

Benefits:

1. Connects to well-known methods.
2. In comparison with conversion, can lead to a more energy efficient network (Li et al. 2022)

Disadvantages:

1. May be computationally intensive to resolve each spatially and thru time

Synaptic Plasticity

Spike-timing-dependent plasticity (STDP) is essentially the most well-known type of synaptic plasticity. Normally, STDP increases the strength of a synapse when a presynaptic (input) spike comes immediately before the postsynaptic spike. Early models have shown promise with STDP on easy unsupervised tasks, although getting it to work well for more complex models and tasks has proven tougher.

Other biological learning mechanisms include the pruning and creation of each neurons and synapses, homeostatic plasticity, neuromodulators, astrocytes, and evolution. There’s even some recent evidence that some primitive kinds of knowledge might be passed down by epigenetics.

Benefits:

1. Unsupervised
2. Can make the most of temporal properties
3. Biologically inspired

Disadvantages:

1. Synaptic plasticity is just not well understood, especially at different timescales
2. Difficult to get to work with non-trivial networks

Evolutionary Optimization

Evolutionary optimization is one other approach that has some cool applications that works well with small networks. Dr. Catherine Schuman is a number one expert and she or he gave a captivating talk on neuromorphic computing to the ICS lab that is obtainable on YouTube.

Benefits:

1. Applicable to many tasks, architectures, and devices.
2. Can learn topology and parameters (requiring less knowledge of the issue).
3. Learns small networks which leads to lower latency.

Disadvantages:

1. Not effective for problems that require deep or large architectures.

Energy Efficiency

Neuromorphic systems have two fundamental benefits: 1) energy efficiency and a pair of) low latency. There are numerous reasons to be excited in regards to the energy efficiency. For instance, Intel claimed that their Loihi 2 Neural Processing Unit (NPU) can use 100 times less energy while being as much as 50 times faster than conventional ANNs. Chris Eliasmith compared the energy efficiency of an SNN on neuromorphic hardware with an ANN with the identical architecture on standard hardware in a presentation available on YouTube. He found that the SNN is 100 times more energy efficient on Loihi in comparison with the ANN on a typical NVIDIA GPU and 20 times more efficient than the ANN on an NVIDIA Jetson GPU. It’s 5–7 times more energy efficient than the Intel Neural Compute Stick (NCS) and NCS 2. At the identical time the SNN achieves a 93.8% accuracy in comparison with the 92.7% accuracy of the ANN.

Neuromorphic chips are more energy efficient and permit complex deep learning models to be deployed on low-energy edge devices. In October 2024, BrainChip introduced the Akida Pico NPU which uses lower than 1 mW of power, and Intel Loihi 2 NPU uses 1 W. That’s so much less power than NVIDIA Jetson modules that use between 10–50 watts which is usually used for embedded ANNs and server GPUs can use around 100 watts.

Comparing the energy efficiency between ANNs and SNNs are difficult because: 1. energy efficiency relies on hardware, 2. SNNs and ANNs can use different architectures, and three. they’re suited to different problems. Moreover, the energy utilized by SNNs scales with the variety of spikes and the variety of time steps, so the variety of spikes and time steps must be minimized to attain the very best energy efficiency.

Theoretical evaluation is usually used to estimate the energy needed by SNNs and ANNs, nevertheless, this doesn’t take note of all the differences between the CPUs and GPUs used for ANNs and the neuromorphic chips for SNNs.

Looking into nature can provide us an idea of what is perhaps possible in the longer term and Mike Davies provided an ideal anecdote in an Intel Architecture All Access YouTube video:

Consider the capabilities of a tiny cockatiel parrot brain, a two-gram brain running on about 50 mW of power. This brain enables the cockatiel to fly at hastens to twenty mph, to navigate unknown environments while foraging for food, and even to learn to control objects as tools and utter human words.

In current neural networks, there may be numerous wasted computation. For instance, a picture encoder takes the identical period of time encoding a blank page as a cluttered page in a “Where’s Waldo?” book. In spiking neural networks, only a few units would activate on a blank page and little or no computation can be used, while a page containing numerous features would fire so much more units and use so much more computation. In real life, there are sometimes regions within the visual field that contain more features and require more processing than other regions that contain fewer features, like a transparent sky. In either case, SNNs only perform work when work must be performed, whereas ANNs rely upon matrix multiplications which can be difficult to make use of sparsely.

This in itself is exciting. A number of deep learning currently involves uploading massive amounts of audio or video to the cloud, where the info is processed in massive data centers, spending numerous energy on the computation and cooling the computational devices, after which the outcomes are returned. With edge computing, you possibly can have safer and more responsive voice recognition or video recognition, you could carry on your local device, with orders of magnitude less energy consumption.

Low Latency

When a pixel receptor of an event camera changes by some threshold, it could possibly send an event or spike inside microseconds. It doesn’t have to wait for a shutter or synchronization signal to be sent. This profit is seen throughout the event-based architecture of SNNs. Units can send events immediately, fairly than waiting for a synchronization signal. This makes neuromorphic computers much faster, by way of latency, than ANNs. Hence, neuromorphic processing is best than ANNs for real-time applications that may profit from low latency. This profit is reduced if the issue allows for batching and you might be measuring speed by throughput since ANNs can make the most of batching more easily. Nevertheless, in real-time processing, reminiscent of robotics or user interfacing, latency is more essential.

All the pieces In every single place All at Once

One in all the challenges is that neuromorphic computing and engineering are progressing at multiple levels at the identical time. The small print of the models rely upon the hardware implementation and empirical results with actualized models guide the event of the hardware. Intel discovered this with their Loihi 1 chips and built more flexibility into their Loihi 2 chips, nevertheless, there’ll all the time be tradeoffs and there are still many advances to be made on each the hardware and software side.

Limited Availability of Business Hardware

Hopefully, this may change soon, but industrial hardware isn’t very available. BrainChip’s Akida was the primary neuromorphic chip to be commercially available, although apparently, it doesn’t even support the usual leaky-integrate and fire (LIF) neuron. SpiNNaker boards was once on the market, which was a part of the EU Human Brain Project but are now not available. Intel makes Loihi 2 chips available to some academic researchers via the Intel Neuromorphic Research Community (INRC) program.

Datasets

The variety of neuromorphic datasets is way lower than traditional datasets and might be much larger. A number of the common smaller computer vision datasets, reminiscent of MNIST (NMNIST, Li et al. 2017) and CIFAR-10 (CIFAR10-DVS, Orchard et al. 2015), have been converted to event streams by displaying the pictures and recording them using event-based cameras. The photographs are collected with movement (or “saccades”) to extend the variety of spikes for processing. With larger datasets, reminiscent of ES-ImageNet (Lin et al. 2021), simulation of event cameras has been used.

The dataset derived from static images is perhaps useful in comparing SNNs with conventional ANNs and is perhaps useful as a part of the training or evaluation pipeline, nevertheless, SNNs are naturally temporal, and using them for static inputs doesn’t make numerous sense if you need to make the most of SNNs temporal properties. A number of the datasets that make the most of these properties of SNNs include:

• DvsGesture (Amir et al. 2017) — a dataset of individuals performing a set of 11 hand and arm gestures
• Bullying10K (Dong et al. 2024) — a privacy-preserving dataset for bullying recognition

Synthetic data might be generated from standard visible camera data without the use of pricy event camera data collections, nevertheless these won’t exhibit the high dynamic range and frame rate that event cameras would capture.

Tonic is an example python library that makes it easy to access a minimum of a few of these event-based datasets. The datasets themselves can take up so much more room than traditional datasets. For instance, the training images for MNIST is around 10 MB, while in N-MNIST, it is nearly 1 GB.

One other thing to take note of is that visualizing the datasets might be difficult. Even the datasets derived from static images might be difficult to match with the unique input images. Also, the good thing about using real data is often to avoid a spot between training and inference, so it will seem that the good thing about using these datasets would rely upon their similarity to the cameras used during deployment or testing.

We’re in an exciting time with neuromorphic computation, with each the investment within the hardware and the advancements in spiking neural networks. There are still challenges for adoption, but there are proven cases where they’re more energy efficient, especially standard server GPUs while having lower latency and similar accuracy as traditional ANNs. A number of firms, including Intel, IBM, Qualcomm, Analog Devices, Rain AI, and BrainChip have been investing in neuromorphic systems. BrainChip is the primary company to make their neuromorphic chips commercially available while each Intel and IBM are on the second generations of their research chips (Loihi 2 and NorthPole respectively). There also seems to have been a specific spike of successful spiking transformers and other deep spiking neural networks within the last couple of years, following the Spikformer paper (Zhou et al. 2022) and the SEW-ResNet paper (Fang et al. 2021).

• Amir, A., Taba, B., Berg, D., Melano, T., McKinstry, J., Di Nolfo, C., Nayak, T., Andreopoulos, A., Garreau, G., Mendoza, M., Kusnitz, J., Debole, M., Esser, S., Delbruck, T., Flickner, M., & Modha, D. (2017). A Low Power, Fully Event-Based Gesture Recognition System. 7243–7252. https://openaccess.thecvf.com/content_cvpr_2017/html/Amir_A_Low_Power_CVPR_2017_paper.html
• Bohte, S. M. (2011). Error-Backpropagation in Networks of Fractionally Predictive Spiking Neurons. In Artificial Neural Networks and Machine Learning https://doi.org/10.1007/978-3-642-21735-7_8
• Dong, Y., Li, Y., Zhao, D., Shen, G., & Zeng, Y. (2023). Bullying10K: A Large-Scale Neuromorphic Dataset towards Privacy-Preserving Bullying Recognition. Advances in Neural Information Processing Systems, 36, 1923–1937.
• Fang, W., Yu, Z., Chen, Y., Huang, T., Masquelier, T., & Tian, Y. (2021). Deep Residual Learning in Spiking Neural Networks. Advances in Neural Information Processing Systems, 34, 21056–21069. https://proceedings.neurips.cc/paper/2021/hash/afe434653a898da20044041262b3ac74-Abstract.html
• Li, C., Ma, L., & Furber, S. (2022). Quantization Framework for Fast Spiking Neural Networks. Frontiers in Neuroscience,16. https://doi.org/10.3389/fnins.2022.918793
• Li, H., Liu, H., Ji, X., Li, G., & Shi, L. (2017). CIFAR10-DVS: An Event-Stream Dataset for Object Classification. Frontiers in Neuroscience, 11. https://doi.org/10.3389/fnins.2017.00309
• Lin, Y., Ding, W., Qiang, S., Deng, L., & Li, G. (2021). ES-ImageNet: A Million Event-Stream Classification Dataset for Spiking Neural Networks. Frontiers in Neuroscience, 15. [https://doi.org/10.3389/fnins.2021.726582](https://doi.org/10.3389/fnins.2021.726582
• Neftci, E. O., Mostafa, H., & Zenke, F. (2019). Surrogate Gradient Learning in Spiking Neural Networks: Bringing the Power of Gradient-Based Optimization to Spiking Neural Networks. IEEE Signal Processing Magazine. https://doi.org/10.1109/MSP.2019.2931595
• Orchard, G., Jayawant, A., Cohen, G. K., & Thakor, N. (2015). Converting Static Image Datasets to Spiking Neuromorphic Datasets Using Saccades. Frontiers in Neuroscience, 9. https://doi.org/10.3389/fnins.2015.00437
• Schuman, C. D., Kulkarni, S. R., Parsa, M., Mitchell, J. P., Date, P., & Kay, B. (2022). Opportunities for neuromorphic computing algorithms and applications. Nature Computational Science,2(1), 10–19. https://doi.org/10.1038/s43588-021-00184-y
• Sengupta, A., Ye, Y., Wang, R., Liu, C., & Roy, K. (2019). Going Deeper in Spiking Neural Networks: VGG and Residual Architectures. Frontiers in Neuroscience, 13. https://doi.org/10.3389/fnins.2019.00095
• Zhou, Z., Zhu, Y., He, C., Wang, Y., Yan, S., Tian, Y., & Yuan, L. (2022, September 29). Spikformer: When Spiking Neural Network Meets Transformer. The Eleventh International Conference on Learning Representations. https://openreview.net/forum?id=frE4fUwz_h

Talks on Youtube