Open Models Almanac
Methodsdata curationtokenization

single mention · data curation

Unigram-objective merge selection

The Unigram training objective is used to select which merge to add to the vocabulary.

The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See tokenization for the methods filed beside it.

Filed under data curation › tokenization