Kolibri-1
Aleph AlphaWe trained the tokenizer with a new algorithm, UniBPE, that respects the morphology of languages better than existing approaches, especially the compound structure of German.
3 more from this model
We combine those two, keeping the bottom-up approach of BPE and using the Unigram training objective for selecting which merge to add to the vocabulary.
We introduce a new way to train tokenizers, UniBPE, that respects the morphology of languages better than existing approaches
it keeps the greedy bottom-up approach of BPE ... and scores merges with the Unigram loss objective