On this planet of natural language processing (NLP), the pursuit of constructing larger and more capable language models has been a driving force behind many recent advancements. Nonetheless, as these models grow in size, the computational requirements for training and inference turn out to be increasingly demanding, pushing against the boundaries of obtainable hardware resources.

Enter Mixture-of-Experts (MoE), a method that guarantees to alleviate this computational burden while enabling the training of larger and more powerful language models. On this technical blog, we’ll delve into the world of MoE, exploring its origins, inner workings, and its applications in transformer-based language models.

The Origins of Mixture-of-Experts

The concept of Mixture-of-Experts (MoE) will be traced back to the early Nineteen Nineties when researchers explored the concept of conditional computation, where parts of a neural network are selectively activated based on the input data. One in all the pioneering works on this field was the “Adaptive Mixture of Local Experts” paper by Jacobs et al. in 1991, which proposed a supervised learning framework for an ensemble of neural networks, each specializing in a special region of the input space.

The core idea behind MoE is to have multiple “expert” networks, each liable for processing a subset of the input data. A gating mechanism, typically a neural network itself, determines which expert(s) should process a given input. This approach allows the model to allocate its computational resources more efficiently by activating only the relevant experts for every input, relatively than employing the complete model capability for each input.

Through the years, various researchers explored and prolonged the concept of conditional computation, resulting in developments similar to hierarchical MoEs, low-rank approximations for conditional computation, and techniques for estimating gradients through stochastic neurons and hard-threshold activation functions.

Mixture-of-Experts in Transformers

While the concept of MoE has been around for a long time, its application to transformer-based language models is comparatively recent. Transformers, which have turn out to be the de facto standard for state-of-the-art language models, are composed of multiple layers, each containing a self-attention mechanism and a feed-forward neural network (FFN).

The important thing innovation in applying MoE to transformers is to exchange the dense FFN layers with sparse MoE layers, each consisting of multiple expert FFNs and a gating mechanism. The gating mechanism determines which expert(s) should process each input token, enabling the model to selectively activate only a subset of experts for a given input sequence.

One in all the early works that demonstrated the potential of MoE in transformers was the “Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer” paper by Shazeer et al. in 2017. This work introduced the concept of a sparsely-gated MoE layer, which employed a gating mechanism that added sparsity and noise to the expert selection process, ensuring that only a subset of experts was activated for every input.

Since then, several other works have further advanced the applying of MoE to transformers, addressing challenges similar to training instability, load balancing, and efficient inference. Notable examples include the Switch Transformer (Fedus et al., 2021), ST-MoE (Zoph et al., 2022), and GLaM (Du et al., 2022).

Advantages of Mixture-of-Experts for Language Models

The first good thing about employing MoE in language models is the flexibility to scale up the model size while maintaining a comparatively constant computational cost during inference. By selectively activating only a subset of experts for every input token, MoE models can achieve the expressive power of much larger dense models while requiring significantly less computation.

For instance, consider a language model with a dense FFN layer of seven billion parameters. If we replace this layer with an MoE layer consisting of eight experts, each with 7 billion parameters, the whole variety of parameters increases to 56 billion. Nonetheless, during inference, if we only activate two experts per token, the computational cost is comparable to a 14 billion parameter dense model, because it computes two 7 billion parameter matrix multiplications.

This computational efficiency during inference is especially useful in deployment scenarios where resources are limited, similar to mobile devices or edge computing environments. Moreover, the reduced computational requirements during training can result in substantial energy savings and a lower carbon footprint, aligning with the growing emphasis on sustainable AI practices.

Challenges and Considerations

While MoE models offer compelling advantages, their adoption and deployment also include several challenges and considerations:

1. Training Instability: MoE models are known to be more liable to training instabilities in comparison with their dense counterparts. This issue arises from the sparse and conditional nature of the expert activations, which may result in challenges in gradient propagation and convergence. Techniques similar to the router z-loss (Zoph et al., 2022) have been proposed to mitigate these instabilities, but further research continues to be needed.
2. Finetuning and Overfitting: MoE models are likely to overfit more easily during finetuning, especially when the downstream task has a comparatively small dataset. This behavior is attributed to the increased capability and sparsity of MoE models, which may result in overspecialization on the training data. Careful regularization and finetuning strategies are required to mitigate this issue.
3. Memory Requirements: While MoE models can reduce computational costs during inference, they often have higher memory requirements in comparison with dense models of comparable size. It is because all expert weights should be loaded into memory, though only a subset is activated for every input. Memory constraints can limit the scalability of MoE models on resource-constrained devices.
4. Load Balancing: To realize optimal computational efficiency, it’s crucial to balance the load across experts, ensuring that no single expert is overloaded while others remain underutilized. This load balancing is often achieved through auxiliary losses during training and careful tuning of the capability factor, which determines the utmost variety of tokens that will be assigned to every expert.
5. Communication Overhead: In distributed training and inference scenarios, MoE models can introduce additional communication overhead on account of the necessity to exchange activation and gradient information across experts residing on different devices or accelerators. Efficient communication strategies and hardware-aware model design are essential to mitigate this overhead.

Despite these challenges, the potential advantages of MoE models in enabling larger and more capable language models have spurred significant research efforts to handle and mitigate these issues.

Example: Mixtral 8x7B and GLaM

For instance the sensible application of MoE in language models, let’s consider two notable examples: Mixtral 8x7B and GLaM.

Mixtral 8x7B is an MoE variant of the Mistral language model, developed by Anthropic. It consists of eight experts, each with 7 billion parameters, leading to a complete of 56 billion parameters. Nonetheless, during inference, only two experts are activated per token, effectively reducing the computational cost to that of a 14 billion parameter dense model.

Mixtral 8x7B has demonstrated impressive performance, outperforming the 70 billion parameter Llama model while offering much faster inference times. An instruction-tuned version of Mixtral 8x7B, called Mixtral-8x7B-Instruct-v0.1, has also been released, further enhancing its capabilities in following natural language instructions.

One other noteworthy example is GLaM (Google Language Model), a large-scale MoE model developed by Google. GLaM employs a decoder-only transformer architecture and was trained on an enormous 1.6 trillion token dataset. The model achieves impressive performance on few-shot and one-shot evaluations, matching the standard of GPT-3 while using only one-third of the energy required to coach GPT-3.

GLaM’s success will be attributed to its efficient MoE architecture, which allowed for the training of a model with an unlimited variety of parameters while maintaining reasonable computational requirements. The model also demonstrated the potential of MoE models to be more energy-efficient and environmentally sustainable in comparison with their dense counterparts.

The Grok-1 Architecture

GROK MIXTURE OF EXPERT

Grok-1 is a transformer-based MoE model with a novel architecture designed to maximise efficiency and performance. Let’s dive into the important thing specifications:

1. Parameters: With a staggering 314 billion parameters, Grok-1 is the biggest open LLM so far. Nonetheless, because of the MoE architecture, only 25% of the weights (roughly 86 billion parameters) are energetic at any given time, enhancing processing capabilities.
2. Architecture: Grok-1 employs a Mixture-of-8-Experts architecture, with each token being processed by two experts during inference.
3. Layers: The model consists of 64 transformer layers, each incorporating multihead attention and dense blocks.
4. Tokenization: Grok-1 utilizes a SentencePiece tokenizer with a vocabulary size of 131,072 tokens.
5. Embeddings and Positional Encoding: The model features 6,144-dimensional embeddings and employs rotary positional embeddings, enabling a more dynamic interpretation of information in comparison with traditional fixed positional encodings.
6. Attention: Grok-1 uses 48 attention heads for queries and eight attention heads for keys and values, each with a size of 128.
7. Context Length: The model can process sequences as much as 8,192 tokens in length, utilizing bfloat16 precision for efficient computation.

Performance and Implementation Details

Grok-1 has demonstrated impressive performance, outperforming LLaMa 2 70B and Mixtral 8x7B with a MMLU rating of 73%, showcasing its efficiency and accuracy across various tests.

Nonetheless, it is vital to notice that Grok-1 requires significant GPU resources on account of its sheer size. The present implementation within the open-source release focuses on validating the model’s correctness and employs an inefficient MoE layer implementation to avoid the necessity for custom kernels.

Nonetheless, the model supports activation sharding and 8-bit quantization, which may optimize performance and reduce memory requirements.

In a remarkable move, xAI has released Grok-1 under the Apache 2.0 license, making its weights and architecture accessible to the worldwide community to be used and contributions.

The open-source release features a JAX example code repository that demonstrates the right way to load and run the Grok-1 model. Users can download the checkpoint weights using a torrent client or directly through the HuggingFace Hub, facilitating easy accessibility to this groundbreaking model.

The Way forward for Mixture-of-Experts in Language Models

Because the demand for larger and more capable language models continues to grow, the adoption of MoE techniques is anticipated to achieve further momentum. Ongoing research efforts are focused on addressing the remaining challenges, similar to improving training stability, mitigating overfitting during finetuning, and optimizing memory and communication requirements.

One promising direction is the exploration of hierarchical MoE architectures, where each expert itself consists of multiple sub-experts. This approach could potentially enable even greater scalability and computational efficiency while maintaining the expressive power of huge models.

Moreover, the event of hardware and software systems optimized for MoE models is an energetic area of research. Specialized accelerators and distributed training frameworks designed to efficiently handle the sparse and conditional computation patterns of MoE models could further enhance their performance and scalability.

Moreover, the mixing of MoE techniques with other advancements in language modeling, similar to sparse attention mechanisms, efficient tokenization strategies, and multi-modal representations, could lead on to much more powerful and versatile language models able to tackling a wide selection of tasks.

Conclusion

The Mixture-of-Experts technique has emerged as a strong tool in the hunt for larger and more capable language models. By selectively activating experts based on the input data, MoE models offer a promising solution to the computational challenges related to scaling up dense models. While there are still challenges to beat, similar to training instability, overfitting, and memory requirements, the potential advantages of MoE models when it comes to computational efficiency, scalability, and environmental sustainability make them an exciting area of research and development.

As the sphere of natural language processing continues to push the boundaries of what is feasible, the adoption of MoE techniques is more likely to play a vital role in enabling the subsequent generation of language models. By combining MoE with other advancements in model architecture, training techniques, and hardware optimization, we are able to sit up for much more powerful and versatile language models that may truly understand and communicate with humans in a natural and seamless manner.