Open Models Almanac
Methodsdata curationtokenization

specific method · filed under data curation

UniBPE

UniBPE is a hybrid tokenizer training method that keeps BPE's greedy bottom-up merging but scores each candidate merge with the Unigram loss objective.1 At each step it applies the merge that lowers the Unigram loss the most, ranking candidates by the closed-form loss change rather than BPE's raw pair count.2

Trade-off It improves morphological alignment over plain BPE, but some useful merges are difficult under it, so a final standard-BPE stage is needed to match BPE's compression.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. it keeps the greedy bottom-up approach of BPE (Gage 1994; Sennrich, Haddow, and Birch 2016) and scores merges with the Unigram loss objective (Kudo 2018 — tech report
  2. at each step it applies the merge that lowersL the most, and otherwise runs exactly likeBPE — tech report
  3. some useful merges are difficult under UniBPE — tech report

Also called Hybrid BPE-Unigram tokenizer training, combine those two, UniBPE hybrid tokenizer training method, UniBPE tokenizer training algorithm, UniBPE tokenizer training method.

sources
3
model
1
lab adopt it
1
strongest
used

Documented in

Aleph Alpha

Evidence

4 quotes from 3 sources, one card per model, each opening with its strongest.

How sources treat it
used 4

Kolibri-1

Aleph Alphaadopted3 sources

We trained the tokenizer with a new algorithm, UniBPE, that respects the morphology of languages better than existing approaches, especially the compound structure of German.

usedsoftware implementationin Kolibri tokenizerAleph Alpha
3 more from this model

We combine those two, keeping the bottom-up approach of BPE and using the Unigram training objective for selecting which merge to add to the vocabulary.

usedsoftware implementationin KolibriAleph Alpha

We introduce a new way to train tokenizers, UniBPE, that respects the morphology of languages better than existing approaches

usedsoftware implementationin KolibriAleph Alpha

it keeps the greedy bottom-up approach of BPE ... and scores merges with the Unigram loss objective

useddata curationin KolibriAleph Alpha
↑