single mention · data curation
UniBPE and standard-BPE vocabulary construction
A construction procedure that uses standard BPE for subsequent merges and adds reserved and special tokens.
The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See tokenization for the methods filed beside it.
Filed under data curation › tokenization