single mention · model architecture
Frozen encoders during pre-training
Keeps the vision and audio encoder weights frozen during pre-training.
The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See multimodal architecture for the methods filed beside it.
Filed under model architecture › multimodal architecture
Explainer · D4 · 25 minMultimodal training data: when pixels and sound enter the recipeImages from the first step train both modalities together, at a cost in text tokens; images added later keep a finished text model, which then has to be protectedExplainer · X2 · 32 minHearing, speaking and keeping time: audio, omni models and multimodal positionsA separate speaking model keeps a large model's steps and context for text at the price of a second stack; one backbone is a single model whose every speech frame takes a step and a position; and time can travel in position ids, which cost no tokens, or in text, which is read like any other token