Open Models Almanac
Methodsdata curationtokenization

methods · data curation

tokenization

42 methods filed at this node or below it, from the sources of 15 models.

In this branch 42

Everything filed at this node or below it, with one collapsible heading per child node.

filed here 42

Quantile normalization of robot actions core · 2 sources · 2 quotes
UniBPE used · 3 sources · 4 quotes
Quick Instruction tokens used · 2 sources · 2 quotes
38 single mentions
ChatML core · 1 source · 1 quote
FAST action tokenizer core · 1 source · 1 quote
FAST+ core · 1 source · 4 quotes
German morphology-aware tokenization core · 1 source · 1 quote
Quantile-bounded action discretization core · 1 source · 1 quote
Robot-specific token interface core · 1 source · 1 quote
XTML core · 1 source · 1 quote
128K-token vocabulary used · 1 source · 1 quote
Bilingual English-German tokenizer used · 1 source · 1 quote
Byte-level byte-pair encoding used · 1 source · 1 quote
Column-first DCT-coefficient flattening used · 1 source · 1 quote
DCT-BPE action compression used · 1 source · 1 quote
Discrete action-token representation used · 1 source · 1 quote
Greedy bottom-up BPE merging used · 1 source · 1 quote
MolmoAct2-FAST Tokenizer used · 1 source · 2 quotes
o200k_harmony tokenizer used · 1 source · 1 quote
o200k_harmony vocabulary used · 1 source · 1 quote
Padding actions to 32 dimensions used · 1 source · 1 quote
SentencePiece used · 1 source · 1 quote
Trie-based tokenization used · 1 source · 1 quote
Unigram-objective merge selection used · 1 source · 1 quote
Selectable robot-output styles optional · 1 source · 1 quote
FAST action-token training evaluated · 1 source · 1 quote

By model

Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

Modelmethods
GLM-5.3-Flash usedChat-template image/video placeholder expansion used—
DeepSeek-V4-Flash usedQuick Instruction tokens used—
Alpamayo 1.5 / 2 Super defaultSingle-image tokenization for multi-camera video inputs defaultDiscrete action-token representation used—
DeepSeek-V4-Pro usedQuick Instruction tokens used—
Gemma 4 31B usedSentencePiece usedSeparate end tokens for pretraining and instruction-tuned models used—
Kimi K3 coreXTML core—
Kolibri-1 coreGerman morphology-aware tokenization coreBilingual English-German tokenizer usedGreedy bottom-up BPE merging usedOnline document tokenization with end-of-sequence markers usedUniBPE usedUniBPE and standard-BPE vocabulary construction usedUnigram-objective merge selection used—
LFM3-1.7B coreChatML core128K-token vocabulary used—
Laguna-S-2.1 usedSubtoken-averaged special-token embedding initialization used—
Mistral-Large-4 usedTrie-based tokenization used—
MolmoAct 2 coreQuantile normalization of robot actions coreRobot-specific token interface coreSimilarity-preserving monotonic assignment of BPE symbols to action bins coreDCT-BPE action compression usedMolmoAct2-FAST Tokenizer usedNormalization and 256-bin discretization of continuous robot states usedNormalize, pad, and tokenize continuous robot actions usedPadding tokenizer and language-model head to a multiple of 512 usedSelectable robot-output styles optional—
OpenVLA / OpenVLA-OFT corePer-dimension 256-bin robot-action discretization coreQuantile-bounded action discretization coreReusing least-used tokenizer vocabulary entries for action tokens core—
Qwen3.5-397B-A17B usedByte-level byte-pair encoding used—
gpt-oss-120b usedo200k_harmony tokenizer usedo200k_harmony vocabulary used—
π0.5 coreByte-pair encoding (BPE) for action-token compression coreDiscrete cosine transform (DCT) action compression coreFAST action tokenizer coreFAST+ coreScale-and-round quantization of DCT coefficients core256-bin proprioceptive-state discretization usedColumn-first DCT-coefficient flattening usedPadding actions to 32 dimensions usedQuantile normalization of robot actions usedFAST action-token training evaluatedPer-dimension 256-bin robot-action discretization not used—
↑