Machine Learning | Natural Language Processing | Data Science
In this text we’ll discuss “Speculative Sampling”, a technique that makes text generation faster and cheaper without compromising on performance.
First we’ll discuss a serious problem that’s slowing down modern language models, then we’ll construct an intuitive understanding of how speculative sampling elegantly speeds them up, then we’ll implement speculative sampling from scratch in Python.
Who’s this handy for? Anyone fascinated with natural language processing (NLP), or leading edge AI advancements.
How advanced is that this post? The concepts in this text are accessible to machine learning enthusiasts, and are leading edge enough to interest seasoned data scientists. The code at the top could also be useful to developers.
Pre-requisites: It is likely to be useful to have a cursory understanding of Transformers, OpenAI’s GPT models, or each. For those who end up confused, you may check with either of those articles:
Over the past 4 years OpenAI’s GPT models have grown from 117 million parameters in 2018 to an estimated 1.8 Trillion parameters in 2023. This rapid growth can largely be attributed to the incontrovertible fact that, in language modeling, larger is healthier.