methods · model architecture
multimodal architecture
154 methods filed at this node or below it, from the sources of 32 models.
Explainer · D4 · 25 minMultimodal training data: when pixels and sound enter the recipeImages from the first step train both modalities together, at a cost in text tokens; images added later keep a finished text model, which then has to be protectedExplainer · X1 · 30 minGetting images and video into a decoder: encoders, projectors, and native fusionHow the mapped models turn pixels into decoder positions, when the vision part trains, and how many positions one image takesExplainer · X2 · 32 minHearing, speaking and keeping time: audio, omni models and multimodal positionsA separate speaking model keeps a large model's steps and context for text at the price of a second stack; one backbone is a single model whose every speech frame takes a step and a position; and time can travel in position ids, which cost no tokens, or in text, which is read like any other token
In this branch 154
Everything filed at this node or below it, with one collapsible heading per child node.
filed here 154
134 single mentions
By model
Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.