Open Models Almanac
Methodsdata curationtokenization

single mention · data curation

Similarity-preserving monotonic assignment of BPE symbols to action bins

BPE symbols are assigned monotonically to action bins so adjacent bins map to adjacent symbols.

The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See tokenization for the methods filed beside it.

Filed under data curation › tokenization