Stability AI’s first release, the text-to-image model Stable Diffusion, worked in addition to—if not higher than—closed equivalents akin to Google’s Imagen and OpenAI’s DALL-E. Not only was it free to make use of, however it also ran on a very good home computer. Stable Diffusion did greater than another model to spark the explosion of open-source development around image-making AI last 12 months.  

This time, though, Mostaque wants to administer expectations:  StableLM doesn’t come near matching GPT-4. “There’s still a whole lot of work that should be done,” he says. “It’s not like Stable Diffusion, where immediately you’ve something that’s super usable. Language models are harder to coach.”

One other issue is that models are harder to coach the larger they get. That’s not only right down to the associated fee of computing power. The training process breaks down more often with larger models and wishes to be restarted, making those models even costlier to construct.

In practice there may be an upper limit to the variety of parameters that almost all groups can afford to coach, says Biderman. It is because large models have to be trained across multiple different GPUs, and wiring all that hardware together is complicated. “Successfully training models at that scale is a really recent field of high-performance computing research,” she says.

The precise number changes because the tech advances, but at once Biderman puts that ceiling roughly within the range of 6 to 10 billion parameters. (Compared, GPT-3 has 175 billion parameters; LLaMA has 65 billion.) It’s not an actual correlation, but on the whole, larger models are likely to perform a lot better.   

Biderman expects the flurry of activity around open-source large language models to proceed. But it is going to be centered on extending or adapting a number of existing pretrained models quite than pushing the elemental technology forward. “There’s only a handful of organizations which have pretrained these models, and I anticipate it staying that way for the near future,” she says.

That’s why many open-source models are built on top of LLaMA, which was trained from scratch by Meta AI, or releases from EleutherAI, a nonprofit that is exclusive in its contribution to open-source technology. Biderman says she knows of just one other group prefer it—and that’s in China. 

EleutherAI got its start due to OpenAI. Rewind to 2020 and the San Francisco–based firm had just put out a hot recent model. “GPT-3 was an enormous change for a whole lot of people in how they thought of large-scale AI,” says Biderman. “It’s often credited as an mental paradigm shift by way of what people expect of those models.”