Enterprises that want tokenizer-free multilingual models are increasingly turning to byte-level language models to cut back brittleness in noisy or low-resource text. To tap into that area of interest — and make it practical at scale — the Allen Institute for AI (Ai2) introduced Bolmo, a brand new family of models that leverage its Olmo 3 models by “bytefiying” them and reusing their backbone and capabilities.
The corporate launched two versions, Bolmo 7B and Bolmo 1B, that are “the primary fully open byte-level language model,” in line with Ai2. The corporate said the 2 models performed competitively with — and in some cases surpassed — other byte-level and character-based models.
Byte-level language models operate directly on raw UTF-8 bytes, eliminating the necessity for a predefined vocabulary or tokenizer. This enables them to handle misspellings, rare languages, and unconventional text more reliably — key requirements for moderation, edge deployments, and multilingual applications.
For enterprises deploying AI across multiple languages, noisy user inputs, or constrained environments, tokenizer-free models offer a approach to reduce operational complexity. Ai2’s Bolmo is an try and make that approach practical at scale — without retraining from scratch.
How Bolmo works and the way it was built
Ai2 said it trained the Bolmo models using its Dolma 3 data mix, which helped train its Olmo flagship models, and a few open code datasets and character-level data.
The corporate said its goal “is to offer a reproducible, inspectable blueprint for byteifying strong subword language models in a way the community can adopt and extend.” To fulfill this goal, Ai2 will release its checkpoints, code, and a full paper to assist other organizations construct byte-level models on top of its Olmo ecosystem.
Since training a byte-level model completely from scratch can get expensive, Ai2 researchers as a substitute selected an existing Olmo 3 7B checkpoint to byteify in two stages.
In the primary stage, Ai2 froze the Olmo 3 transformer in order that they only train certain parts, similar to the local encoder and decoder, the boundary predictor, and the language modeling head. This was designed to be “low cost and fast” and requires just 9.8 billion tokens.
The following stage unfreezes the model and trains it with additional tokens. Ai2 said the byte-level approach allows Bolmo to avoid the vocabulary bottlenecks that limit traditional subword models.
Strong performance amongst its peers
Byte-level language models are usually not as mainstream as small language models or LLMs, but it is a growing field in research. Meta released its BLT architecture research last yr, aiming to supply a model that is powerful, processes raw data, and doesn’t depend on fixed vocabularies.
Other research models on this space include ByT5, Stanford’s MrT5, and Canine.
Ai2 evaluated Bolmo using its evaluation suite, covering math, STEM reasoning, query answering, general knowledge, and code.
Bolmo 7B showed strong performance, outperforming character-focused benchmarks like CUTE and EXECUTE, and likewise improving accuracy over the bottom LLM Olmo 3.
Bolmo 7B outperformed models of comparable size in coding, math, multiple-choice QA, and character-level understanding.
Why enterprises may select byte-level models
Enterprises find value in a hybrid model structure, using a mixture of models and model sizes.
Ai2 makes the case that organizations also needs to consider byte-level models not just for robustness and multilingual understanding, but since it “naturally plugs into an existing model ecosystem.”
“A key advantage of the dynamic hierarchical setup is that compression becomes a toggleable knob,” the corporate said.
For enterprises already running heterogeneous model stacks, Bolmo suggests that byte-level models may now not be purely academic. By retrofitting a robust subword model relatively than training from scratch, Ai2 is signaling a lower-risk path for organizations that want robustness without abandoning existing infrastructure.
