The model is in a position to outperform competitors despite being substantially smaller.

I recently began an AI-focused educational newsletter, that already has over 160,000 subscribers. TheSequence is a no-BS (meaning no hype, no news, etc) ML-oriented newsletter that takes 5 minutes to read. The goal is to maintain you up thus far with machine learning projects, research papers, and ideas. Please give it a try by subscribing below:

Coding has been one of the lively areas of development in the inspiration model space. OpenAI opened the floodgates to this space with models like Codex, which eventually morphed into GPT-4. Nonetheless, firms comparable to Amazon and Salesforce have also released incredibly high-quality work on this domain. The premise of coding foundation models has been the power to pre-train a model in numerous code datasets and expect capabilities to surface across different programming languages. Quantity and size over quality has been the mantra of the primary generation of coding language models. Recently, Microsoft Research published a paper with a catchy title: “Textbooks is all You Need” that challenged this assumption by making a small coding language model trained solely in textbook quality datasets. The paper immediately became super popular throughout the LLM community given its unique approach to LLM training producing a model that was significatively smaller but equally performant than alternatives.

Demonstrating the importance of high-quality data, Microsoft Researched launched into training a 1.3B-parameter model, known as phi-1, for about eight passes over 7B tokens (comparable to barely over 50B total tokens observed). Subsequently, the model underwent finetuning using lower than 200M tokens. Their pretraining process involved utilizing “textbook quality” data, comprising each synthetic data generated using GPT-3.5 and filtered content sourced from the online. The…