Biological approaches to understanding artificial intelligence (AI)

Up to now several months, I’m sure everyone’s feed has been flooded with news about AI models comparable to ChatGPT and GPT-4: excitements about models’ emerging intelligence and capabilities; fears of how they’ll replace so many roles by having the ability to write codes, generate arts, and access external tools via APIs (e.g. ChatGPT plugins); skepticisms about their intelligence still lacking world models, and about all those state-of-the-art problem-solving performances are merely a result of information leakage.

On this post, I won’t discuss how one can train, validate, and deploy AI models. Fairly, I’d wish to take a scientific view of a biologist on how one can study AI models. I discovered reviewing classic biological approaches inspires directions in understanding AI models.

Biology is the scientific study of life, most which possess natural intelligence. There are two research directions to check its artificial counterpart: 1) constructing AI by imitating biological intelligence; 2) understanding AI models using biological research strategies.

To construct artificial intelligence, one school of strategies is to mimic the design of natural intelligence, precisely the study of Bionics or biologically-inspired engineering. That is how the prototype of airplane was invented: da Vinci researched the wings of birds and designed a man-powered aircraft in his Codex on the Flight of Birds (1502). Nevertheless, mimicking doesn’t equate copying, it often diverges from its biological counterpart to be more efficient. Fortunately, our modern airplanes don’t flap the wings. Several components of AI models have been invented by inspirations from biology, particularly from animal’s nervous systems:

How can we use biology to know AI? I’ll concentrate on the sub-disciplines inside biology to supply some examples of how one can understand AI models.

Ages ago, a collage student majoring in biology defined foundational biology as sub-disciplines which are generalizable to species across all three domains, which apparently excludes neuron science and immunology.

OpenAI’s GPT models are named after Leonardo da Vinci, famously often called an artist, inventor and polymath. What ChatGPT missed is the indisputable fact that da Vinci can also be one in every of the pioneers in the sphere of anatomy.

The anatomy of AI models is comparable to the hierarchical organization of multicellular organism. Consider a person text-to-image AI model (e.g. DALL-E 2) as an single organism, its text encoder and image decoder are organs with distinct functions. Contained in the organs, there are transformer blocks as tissues, that are composed of the person attention heads, normalization layers as different cells. Inside theses cells, tensors of weights and biases are essentially the biomolecules carrying out the functions.

Comparing to natural organisms, it’s strikingly easy to dissect and observe the anatomy of an open-sourced AI model, as we will directly access the modules and parameters of the model without sacrificing the model or putting it under Leeuwenhoek’s microscope. Nevertheless, it continues to be obligatory to develop tools for us humans to know how the learned parameters work at microscopic scale. Olah et al. (2020) developed such a algorithmic tool able to “zooming into” the neurons inside transformers to know their specialized functions.

Genetics is the study of interaction between (collection of genes) and in organisms. The basic principle of genetics states that genetic information encoded in nucleic acids (DNA/RNA) dictate phenotypes, that are moreover shaped by environments, and that genetic information is passed to offsprings by reproduction.

In AI models, the learnt parameters of neural networks are analogous to their DNA sequences. Model architectures aren’t a part of genotypes because AI models can reproduce smaller “offsprings” with different architectures, via processes like model distillation. Architectures are more just like the nucleus, an optional carrier of DNA.

Phenotypes are the gathering behaviors AI models exhibit in response to environment. In other words, phenotype is what an AI model would output in response to different environmental inputs. Two classical approaches utilized in genetics research are forward and reverse genetics.

involves starting with a phenotype of interest after which identifying the genetic basis for that phenotype by mutagenesis to create random mutations within the genome. A number of a long time ago, it was impossible to mutate any gene of interest in biological systems: mutagenesis can only be performed at random, using technologies comparable to TALEN and chemical mutagens. Fortunately, it is simpler to mutate AI’s genome by simply perturbing the values of its parameters. The mutagenesis may also be done in a guided fashion moderately than randomly: comparable to methods guided by the activation, gradient, or conductance of the neurons. Lots of those attribution methods (e.g. Integrated Gradient, GradCAM) were developed for interpreting deep networks’ predictions: with the output fixed, attribute the predictions to the input features. When the attribution is performed at a finer level, researchers can pinpoint the precise neuron within the model chargeable for certain predictions (i.e. neuron attribution). Using this “forward genetic approach”, Dhamdhe et al. (2019) found specific “genes” in AI models necessary for the prediction for a particular input, or over a set of inputs.

starts from a known gene after which determining the phenotype that results when that gene is altered, using techniques like gene editing (e.g., CRISPR-Cas9). This is definitely commonly utilized by AI researchers: ablation studies for a recent AI model essentially knock out certain components in a model to check the changes in performances on a set of benchmarking tasks. Nevertheless, oftentimes AI researchers only concentrate on one or a couple of phenotypes. It could be helpful to scan the complete phenome and to ablate the model at finer granularity to know on the genetics of AI.

Recently, a novel research direction in AI is in-context learning (ICL), where AI model’s prediction might be improved by providing additional instructions and/or demonstrations, without updating its parameters. ICL is parallel to a comparatively young field of , which is the study of stable changes in phenotypes that don’t involve alterations within the genome. Those prompts prepended to an input is identical to the epigenetic modifications on the genome, able to altering the phenotypes.

In a genetic perspective, one necessary subject is : the passing of genetic information from parents to offsprings. This results in a somewhat scary inquiries to some: how do AI models reproduce? This can also be nice segway to the subsequent discipline.

Evolution is the change in heritable traits of biological populations over successive generations. I consider most AI models evolve via asexual reproduction: a model duplicates itself and updates its parameters to turn into barely more adaptive to the environment. The fitness is governed by some objective function(s). In some special circumstances comparable to adversarial learning (e.g. generative adversarial networks), the educational of AI models could also be considered sexual reproduction. The training technique of AI models using stochastic gradient descent (SGD) on large volume of static data is basically natural selection.

To responsibly harness the facility of AI models, we’d like to tame the AI models to be higher at following human’s instructions. One such technique is Reinforcement Learning from Human Feedback (RLHF). Evolutionarily speaking, that is often called the bogus selection process, identical to how humans domesticated horses and dogs.

There are potentially more analogies we will draw between AI and biology, comparable to ecology and multi-agent RL. I hope the analogies presented in this text are refreshing. We’re probably at some extent in history where understanding the AI models we already built is more necessary than aimlessly training greater models with more data. Because the scientific discipline studying natural intelligence, I think there are potentially more research strategies AI researchers can adopt from biology to understanding AI.