Text Model progress goes strong!
As previously announced, we are actually training our own large language models from scratch.
Since this isn’t a simple endeavor, we’re starting out with a small proof of concept model.
This permits us to discover and fix various issues with the training methodology and data, without losing a considerable amount of time and compute when things go flawed.
To this point, this approach has already paid off and we have now fixed numerous issues in the course of the means of training our small proof of concept model, one among which did necessitate a restart of the training run. The total pretraining run for this small model, speaking from a compute-only perspective, takes about five days on our — only recently accomplished — cluster.
At this cut-off date, we have now finished the pretraining phase with very promising results (and other evals near or beyond GPT-NeoX 20B). We’ve now advanced to tuning the model for a .
During this we identified some further data issues, which are actually addressed, so training could proceed further.
We plan to release this smaller proof of concept model within the near future, so people can get a primary impression of the sort of results our training methodology can achieve, while we start working on a bigger, more powerful model.
That’s all of the news for now… except, we have now a latest mascot introduction coming up:
Ladies and gentlemen, gather ‘round and brace yourselves for a mind-bending encounter like no other!
Prepare to delve into the boundless realms of innovation and imagination as we unveil to you the one, the one… Shoggy! But fear not, for this Shoggoth isn’t of the eldritch horror variety which may send shivers down your spine.
No, no! This Shoggoth, affectionately generally known as “,” is the extraordinary visual representation of NovelAI’s cutting-edge H100 supercompute cluster.
Using Shoggy as a conduit for inspiration, we shape our innovations:
Whether you’re a author searching for the proper plot twist or an artist craving for a stroke of genius, this magnificent amalgamation of information and dreams is here to shape our work into latest fantastical text and image models.
