Open Models Almanac

Data · D4

Multimodal training data: when pixels and sound enter the recipe

Images from the first step train both modalities together, at a cost in text tokens; images added later keep a finished text model, which then has to be protected

Illustration of a long bolt of ink-blue cloth coming off a simple wooden loom. Coral and teal threads run through the cloth from its first rows, a gold thread joins halfway along, and near the far end a few small embroidered stitches sit on top of the finished weave.

DeepSeek V4.1 Flash reads 45T tokens in pre-training, and images are in the mix “from the start of language-model pre-training”. The report gives the corpus as “a 7:1 token ratio of text-only to multimodal data”, so one corpus token in eight belongs to a sample that holds an image (computed: 1/(7+1) = 12.5%). Nemotron 3.5 Super VL goes the other way. Its base model “is pre-trained on text-only data”, and NVIDIA brings in “large quantities of vision data in the SFT stage”, after warming up a vision projector on the pre-trained text base. Both models read images and text.

X1 covers how pixels become positions in a decoder. This article covers the training side: when non-text data enters, its share and unit, what the encoder does meanwhile, what the data is, and what labs measured about the cost to text. Twelve labs in the map (the open-weight models this site tracks) ship a model that takes images or audio, counting Liquid through its LFM3 vision and audio branches. Eleven say at least which stage the non-text data enters; StepFun says nothing about it.

Background

The parts come from X1. An encoder, usually a vision transformer (ViT), turns an image into a grid of vectors. It is often pretrained contrastively, to match images with their captions, as CLIP and SigLIP are. A projector (some labs say connector or adapter), often a two-layer MLP, maps those vectors to the language model’s embedding width, and the results sit in the token sequence beside the text. In the usual setup, which the reports here mostly don’t state, the loss is computed on text positions only. An image position then has no target: image tokens take compute and context, and the encoder’s gradient comes only from text after the image, so an image placed after all of its text sends the encoder nothing from that sample.

The stages are those of the primer. Pre-training is the long first stage on broad data. Mid-training is a shorter stage on a finished base with changed data, and labs use the word for several operations (S2). Supervised fine-tuning (SFT) trains on conversations, and reinforcement learning (RL) on tasks with a reward. To freeze a part is to exclude its weights from the update. A warm-up here means a short stage that trains only a new part, such as the projector, while the rest stays frozen.

Multimodal training adds kinds of count. Corpus tokens are what a lab’s mix holds, and whether epochs are included varies. Tokens of multimodal samples include the text in those samples: a captioned image or an interleaved web page counts all its tokens on the multimodal side. Image tokens are only the image positions. Some labs count samples or documents instead, and audio comes in hours. A ratio in one of these units can’t be read in another.

Training recipes aren’t in configs or code, so the evidence is the labs’ technical reports first, then model cards and official blogs. We count labs, out of the 12 above. Robot models in the map (OpenVLA, π0.5, GR00T and others) train a vision-language model on robot trajectories, a different data problem that this article leaves out.

Where non-text data enters

Sorted by the stage at which non-text data first enters, the labs fall into five groups, plus four models that put images in pre-training without saying when. The grouping is ours; each row’s facts are quoted. Some rows cite a predecessor or sibling of a mapped model, whose report gives detail the mapped model’s card leaves out: Kimi K2.5 (for K3), Nemotron 3 Nano Omni (for 3.5 Super VL), Qwen3.5-Omni (for Qwen3.5) and LFM2-VL (for Liquid’s LFM3, whose report describes text training only, so LFM2’s schedule shouldn’t be read onto it).

Where non-text data entersModelsWhat the lab says
First step of pre-trainingKimi K3, DeepSeek V4.1 Flash, Mistral Large 4, MiniMax M3“jointly optimized from the start of training”; “from the start of language-model pre-training”; “We train from scratch on interleaved image–text data”; “mixed-modality training from Step 0”
Pre-training, start not statedQwen3.5, Gemma 4, Inkling, GLM-5.3-Flash“Early fusion training on trillions of multimodal tokens”; “web documents, code, images, and audio (for E2B, E4B and 12B)”; “45 trillion tokens of text, images, audio and video”; “30T-token multimodal pre-training corpus”
A text stage, then a mixed stage, inside pre-trainingMiMo-V2.6, Kimi K2.5MiMo-V2.6-Flash 26T text then 22T “omni-modal”, -Pro 27T then 3T; K2.5 “continues from a near-end Kimi K2 checkpoint over additional 15T vision-text tokens”
After language mid-trainingLFM2-VLconnector pre-training, then multimodal mid-training for about 50B tokens, then SFT
SFTNemotron 3.5 Super VL, Nemotron 3 Nano Omnia projector warm-up, then two SFT stages; seven SFT stages that “progressively introduce new modalities”
Continued training of a finished modelDeepSeek-V4-Flash-Vision-Exp; Qwen3.5-Omni (audio, on a vision-language base)“incorporating visual modules and undergoing continued training to unlock visual understanding capabilities”; “the LLM component of Qwen3.5-Omni is initialized with parameters from Qwen3.5”, locked at first

Stage names follow each lab’s vocabulary, not the volume. Nano Omni’s seven “SFT” stages hold 466.9B tokens, prompts and repeats included, about nine times LFM2-VL’s multimodal mid-training (computed). One of them, 100.5B tokens of speech, sound and music, trains the audio projector and encoder with the language model frozen. Liquid’s audio model, LFM2-Audio, trains in its own three phases (alignment, mid-training, post-training) on the LFM2-1.2B backbone, from a checkpoint the report doesn’t name.

Qwen3.5 sits in the second row because “early fusion” is Qwen’s word (README and cards) and none of its documents gives a step. The labs’ labels don’t separate the rows. DeepSeek calls its recipe “native multimodal training”, and Moonshot calls K3 “natively multimodal”. Xiaomi calls MiMo-V2.5, V2.6’s predecessor, which also trains text first, “a native omnimodal model”, and Qwen3.5-Omni is “Natively pretrained in an omnimodal manner” on a finished model. Liquid’s report lists “Native multimodality” among its contributions for a recipe that adds vision after language mid-training. Moonshot called Kimi K2.5 “a native multimodal model built upon Kimi K2”, a near-finished text model, and uses the same word for K3, which trains on images “from the start of training”.

Images in pre-training, start not stated: Qwen3.5 (“Early fusion training on trillions of multimodal tokens”); Gemma 4 (“web documents, code, images, and audio (for E2B, E4B and 12B)”); Inkling (“45 trillion tokens of text, images, audio and video”); GLM-5.3-Flash (“30T-token multimodal pre-training corpus”). Hover a cell for its quote.

Figure 1.Where non-text data first enters, by model. Grey is text-only training, colour is training with images or audio, and dashed cells are stages whose data the sources don't describe. Stages are drawn at equal width; this is an order, not a token axis (X1's fusion timeline has the tokens).

What the timing costs

None of the evidence below is a head-to-head comparison on a shipped model.

Text tokens, by arithmetic

Mistral gives the one statement about a step-0 recipe’s cost to text, in its report’s data section: “With careful dataset design, we observe no regret in text perplexity when matching the number of text tokens seen; while matching the total training horizon yields modest regret.” Regret means worse than a text-only run. The report gives no numbers, measures perplexity only, and doesn’t say whether the comparison ran on the full model or a smaller proxy.

Let pp be the non-text share of the tokens seen. To see as many text tokens as a text-only run, the multimodal run needs 1/(1−p)1/(1-p) times the total, or p/(1−p)p/(1-p) more. At an equal total, it sees a fraction pp fewer text tokens, and a model’s text skill grows with the text it sees. Mistral doesn’t state its pp. At p=12.5%p = 12.5\%, DeepSeek’s multimodal-sample share used here only for scale, matching text tokens costs 14.3% more tokens, and matching the total gives up 12.5% of the text (computed). DeepSeek’s 12.5% includes the text inside its multimodal samples, so its own non-text share is lower.

Read together, Mistral’s two halves say the images neither helped nor hurt text perplexity at its scale, and the regret at an equal total is what seeing fewer text tokens would cost on its own (our reading). The run with equal text tokens is also longer, so it takes more optimizer steps. In classical terms, training on both from the start is multi-task learning, whose risk is that the tasks interfere, and adding a modality later is sequential transfer, whose risk is forgetting. Aghajanyan et al. (2023) measured how modalities compete in mixed-modal models of 8M to 30B parameters over discrete tokens, and trained a 30B speech-text model that “significantly outperforms the corresponding unimodal models”.

Toy arithmetic; Mistral's share is not stated.

Non-text share p = 12.5%.

Same text tokens: the multimodal run is 1.143× as long (+14.3% tokens).

Same total: it sees 12.5% fewer text tokens.

Mistral: “With careful dataset design, we observe no regret in text perplexity when matching the number of text tokens seen; while matching the total training horizon yields modest regret.” Grey is text, blue non-text; the axis is in units of the text-only run's total tokens, from 0.

Figure 2.Two ways to compare a multimodal run with a text-only one. Holding text tokens equal makes the run longer by p/(1−p); holding the total equal takes p of the text away. Mistral reports no regret in the first case and modest regret in the second, without numbers; the share p here is a slider, not Mistral's figure.

One controlled comparison, on an earlier model

Moonshot’s report on Kimi K2.5, an earlier model not in the map, has the only ablation we found that moves the point where vision enters (§2.1, Table 1). It held “the total vision and text token budgets fixed” and trained text-only “for a specifically calculated number of tokens before introducing vision data”. Each arm puts 10% of all its tokens into vision: from the start at a 10% share, from halfway at 20% (0.5 × 20%), from 80% of the way at 50% (0.2 × 50%), computed from the table’s columns.

Vision enters atVision share afterVision knowledgeVision reasoningOCRText knowledgeText reasoningCode
0% (early)10%25.843.865.745.558.524.8
50% (mid)20%25.040.764.143.958.624.0
80% (late)50%24.239.061.543.157.824.0

Early beats late on all six columns, by 1.6 to 4.8 points on the vision ones and 0.7 to 2.4 on text and code, and beats mid on five; mid leads text reasoning by 0.1 (computed). Moonshot’s caption: “Early fusion with a lower vision ratio yields better results given a fixed total vision-text token budget.” Its appendix describes the text side: “when vision data is first introduced, text capability initially degrades before gradually recovering”, a pattern it calls “dip-and-recover” and continual-learning research calls the stability gap (De Lange et al., 2022).

The model size and budget aren’t given, and each arm is one run without error bars, so gaps under a point can’t be told from run-to-run noise. The arms lie on one line (share × remaining fraction = 10%), so timing and vision share can’t be separated. Text tokens are equal in all three, so the text gaps measure interference or ordering, not the cost in text tokens, and there is no text-only arm. The late arm puts its vision, and its thinnest text, in the last fifth of training, and may still be recovering from its dip when scored. “Late” still sits inside pre-training, so the ablation says nothing about attaching vision to a finished model. Moonshot’s shipped K2.5 itself took a later route, adding vision to a near-end K2 checkpoint; with K3 it moved to step 0.

Text scores against the model’s own backbone

Two labs set a multimodal model’s text scores beside one of their own models: NVIDIA beside the backbone it started from, Qwen beside a post-trained sibling. Nemotron 3 Nano Omni’s Table 10 compares Omni with its backbone, Nemotron 3 Nano 30B-A3B. AIME25 falls from 89.1 to 82.1, LiveCodeBench from 68.3 to 63.2 and MMLU-Pro from 78.3 to 77.3; IFBench rises from 71.5 to 74.2 and AA-LCR, a long-context test, from 35.9 to 41.0. Omni is lower on five of eight (computed), which by itself shows no direction, and AIME25’s 7 points are about two of its 30 problems. Omni had its own SFT and RL, so this isn’t vision data alone, and no row has an interval. Qwen’s report calls Qwen3.5-Omni-Plus “on par with its text-only counterpart”, Qwen3.5-Plus-Instruct. In its Table 4, Omni-Plus is lower on 12 of 14 benchmarks, equal on one and higher on one (computed), a consistent direction, with the largest gap on AA-LCR: 57.0 for Omni-Plus against 62.0. That base already had vision, so the comparison is about adding audio and Omni’s own post-training.

The other statements compare with something else. LFM2-VL-3B keeps “strong language capabilities” against other vision-language models (SmolVLM2-2.2B, Qwen2.5-VL-3B), not its own backbone. DeepSeek’s API change log calls Vision-Exp “on par with the official DeepSeek-V4-Flash” on pure-text tasks and prints no V4-Flash numbers beside it.

Throughput

Model FLOPs utilisation (MFU) is the share of the hardware’s peak arithmetic a run achieves. Mistral reports “sustaining 99% of text-only MFU” by prefetching images and caching their encoder features. Qwen3.5’s card claims “Near-100% multimodal training efficiency compared to text-only training”, and Kimi K2.5 reached “a multimodal training efficiency of 90% relative to text-only training”, neither defining the measure. For K3, “most of the ViT computation is hidden within pipeline bubbles”, the idle gaps when pipeline-parallel stages wait for one another. These are utilisation ratios, not costs per sample: an image position still costs a backbone forward and backward pass like a text token. Read at face value, they say the encoder and data loading add little on top, so images cost about what their share of the token budget costs.

Mix ratios, and why they don’t share an axis

Five labs print a number for how much of their training is multimodal, and each counts a different thing.

DeepSeek’s 7:1 is a ratio of corpus tokens between two kinds of sample. Where a document existed in both forms, DeepSeek replaced the text-only version with the multimodal one at the larger of their two epoch counts, and gives the ratio after that, so it most likely counts repeats. The multimodal side holds the text of captions and interleaved pages, so image tokens are under one in eight, by an amount not stated. Qwen3.5-Omni’s second pre-training stage lists “approximately 4 trillion tokens”: text 0.92T, audio 1.99T, image 0.95T, video 0.14T and video-audio 0.29T. Those parts add up to 4.29T, which is 0.29T over the stated total, exactly the video-audio part, and the report doesn’t say how a mixed sample’s tokens are bucketed. On the parts’ sum, text is 21.4% (computed). MiMo-V2.6 gives stage sizes, not modality shares: the omni stage is 45.8% of Flash’s 48T and 10.0% of Pro’s 30T (computed), and those stage tokens include text.

The other two count positions of different kinds within one stage. LFM2-Audio’s mid-training “sees around 100 billion tokens, of which approximately 62% are text tokens, 25% are audio output frames (8-codebook Mimi tokens), and 13% are audio input embeddings”. Those are three units. Output audio is generated as discrete codes: Mimi, a neural audio codec, turns each short frame of sound into eight codes, one from each of eight codebooks. Input audio enters as one continuous vector per frame, with no code at all. Nemotron 3 Nano Omni’s stage 4, the first to train on all modalities, gives shares of its 57.3B tokens: vision 53.4%, audio 16.9%, short video 11.0%, text 6.1%, and smaller parts. Its stage table counts tokens including prompt tokens, which carry no loss (“masked tokens from the prompt”), and “any sample repetitions”.

Even a share of image tokens would move with settings the mixes don’t show. Resolution, tiling and frame count set how many positions an image or video takes, and they change by stage. LFM2-VL’s images take “from 128 to 256 for untiled images to around 2,800 tokens for heavily tiled inputs”, and its mid-training disables “high-resolution tiling” (up to 4 tiles in some OCR runs) that its SFT enables. DeepSeek trains its encoder’s contrastive stage at up to 224 by 224 pixels and everything after it between 544 by 544 and 1344 by 1344. Nano Omni samples up to 64 video frames in its stages 1 and 4 and 256 in the last two.

Image tokens: a part of the coloured share, not stated.

How a mixed sample's tokens are bucketed isn't stated.

The omni stage includes text; its non-text share isn't stated.

An output frame stands for 8 codebook entries; an input embedding is one continuous vector.

The printed shares sum to 99.94%.

Grey is text-only data; colour is the stage the numbers belong to (blue pre-training, teal mid-training, violet SFT). Read each panel alone: no two share a scale. Hover a bar for its quote.

Figure 3.Five labs' mix numbers, each in its own unit: corpus tokens by sample kind, tokens by modality bucket, tokens by stage, positions of three different kinds, and tokens with prompts and repeats. Read each panel alone.

The published ablation is older and outside the map. MM1 (McKinzie et al., 2024) varied captioned images, interleaved image-text documents and text-only data, with a base mix of 45%, 45% and 10%, counted in documents. Few-shot means the prompt holds a few worked examples, zero-shot none. MM1’s “Data Lesson 1” reads: “Interleaved data is instrumental for few-shot and text-only performance, while captioning data lifts zero-shot performance.” The authors kept about that balance, a caption:interleaved:text ratio of 5:5:1. No mapped lab publishes such a sweep of its own mix.

The mix matters to a mixture-of-experts (MoE) router too. DeepSeek routes each token to experts by a score plus a per-expert bias that it nudges to even out the load, and keeps “separate expert-wise correction biases for text and image tokens”, since the two “may induce different expert-routing preferences” (per-modality balancing; see C3). DeepSeek is the only lab that describes one.

Easing the new data in

When a modality is added later, the model already works, and the risk is that training on the new data erases some of what the old data taught, which is called catastrophic forgetting. Flamingo (Alayrac et al., 2022) saw it in an ablation: against its recipe with the language model frozen, “fine-tuning our pretrained LM also leads to a drop in performance of −8.0%” on the overall few-shot vision-language score of its ablation model, which the authors attribute to forgetting. They add: “In our setting, freezing the language model is a better alternative to training with the pre-training dataset (MassiveText) in the mixture.”

LLaVA (Liu et al., 2023) set the template. Its first stage keeps “both the visual encoder and LLM weights frozen” and trains only the projection, on 595K image-text pairs; the second trains the projector and the language model together. Our reading of why the order helps follows LP-FT (Kumar et al., 2022), which showed for a new output head that training the head first, then everything, keeps pretrained features from being distorted. A new projector starts random and puts images in coordinates the backbone has never seen, so the loss on image samples is high. Trained together at once, the large early gradients would reshape the backbone to fit the projector; trained alone first, the projector moves toward the backbone instead.

The labs’ versions differ in what they freeze and for how long. LFM2-VL trains its connector first, with “the image encoder and language backbone frozen”, on a captioning corpus. LFM2-Audio does the same with speech-recognition data, and moves on “when the loss plateaus”. Xiaomi’s blog lists “projector warmup to align the audio and visual projectors with the language model” between MiMo-V2.5’s text and multimodal pre-training. Moonshot aligned K2.5’s encoder with a smaller LLM, Moonlight-16B-A3B, through a caption loss over about 1T tokens, then ran a “very short second stage” that updates “only the MLP projector” before joint pre-training. Qwen3.5-Omni locks the language model in its first stage, and its encoders start by “training their respective adapters before training the encoders”. Nemotron 3 Nano Omni trains only the vision projector first, then unfreezes the language model and vision encoder, and repeats the pattern for audio with the language model frozen before a stage that trains everything. In its later text-only RL it freezes “the LM input token embedding parameters”. NVIDIA’s reason: “This staged approach mitigates catastrophic forgetting and stabilizes cross-modal alignment during training.” No lab reports what its warm-up gains; Liquid’s one ablation, below, found its own unnecessary once the mix is ramped.

Annealing the mix

LFM2-VL adds a second protection inside its multimodal mid-training: “We apply a data annealing schedule that starts with 100% text-only data and, over the first 5% of steps, linearly decreases its fraction to 30%, which is then kept fixed for the remainder of training.” “Annealing” is Liquid’s word for this ramp of the mix; the learning rate isn’t involved. The ramp is a curriculum on the mix, familiar data first, and a soft warm-up too: image samples, which bring the new connector’s large early gradients, start at none. The 30% of text kept for the other 95% of steps is the older remedy for forgetting, rehearsal: keep training on some of the old data. The fraction is given per step, and the report doesn’t say whether it counts samples or tokens.

Steps seen: 100% of the stage. Text-only fraction now: 30.0%.

Average so far: 31.75% text-only, 68.25% multimodal.

Grey: text-only data; teal: multimodal data, shaded up to the marker. Both axes linear from 0. Liquid: “We apply a data annealing schedule that starts with 100% text-only data and, over the first 5% of steps, linearly decreases its fraction to 30%, which is then kept fixed for the remainder of training.” The token scale assumes equal tokens per step, which the report doesn't state.

Figure 4.LFM2-VL's text-only share during multimodal mid-training. It falls from 100% to 30% over the first 5% of steps and then holds, so averaged over the stage the data are 31.75% text-only (computed). Whether the share counts samples or tokens isn't stated.

Averaged over the ramp, the data are 65% text-only, and over the whole stage 31.75% (computed: 0.05×65%+0.95×30%0.05 \times 65\% + 0.95 \times 30\%). Liquid reports, without numbers, model size or metric: “Ablations showed that the annealing data schedule removes the need for connector pre-training, though we kept the connector pre-training stage as it did not hurt performance.” The same stage trains every part, “with a learning rate ratio of 5:5:1 for the text backbone, connector, and encoder”, so the encoder learns at a fifth of the backbone’s rate, a cheap way to keep a pretrained part near where it started. These figures are for LFM2-VL-3B, and the shipped 3B is a weight-space merge of several candidate runs that varied, among other things, “curriculum schedules”, so its weights don’t come from this one schedule alone.

A warm-up adds a stage with its own data and its own stopping rule. A frozen part can’t adapt: nothing after a frozen encoder can recover what its features discard, and what they keep weakly is hard to read out. Rehearsal and a ramp spend steps on text the model already knows. None of the labs reports what these protections saved.

The encoder across the stages

X1 tabulates where each encoder comes from. Read as a schedule across stages, the labs’ choices run from never frozen to absent.

Encoder scheduleModelsDetail
Never frozenKimi K3MoonViT-V2 trained “entirely from scratch with next-token prediction”, jointly from step 0
Frozen until the learning rate decaysDeepSeek V4.1frozen “until the learning-rate decay stage”, from 28T of 45T tokens, with its final norm layer and the projector trainable; then “a smaller learning rate”
Frozen through pre-training, trained in SFTMistral Large 4its own 1.6B encoder, pretrained “with a captioning objective” over 8T tokens
Frozen in pre-training, later not statedGemma 4 31B and the other sizes with an encoder
Pretrained, then trained jointly; no freeze statedMiMo-V2.6in-house ViT trained through a small LLM’s cross-entropy on over 4T image tokens, then “jointly train the full model”
Frozen for a projector stage, then trainedLFM2-VL, Nemotron, Qwen3.5-OmniLFM2-VL at a fifth of the backbone’s rate; Nemotron in SFT; Qwen’s encoders on a locked language model first
No encoderGemma 4 12B, Inklingpatches mapped to embeddings by a projection or a small per-patch MLP (encoder-free)

With the loss on text alone, K3’s encoder learns only what helps predict the text that follows. A frozen encoder needs no backward pass, so its forward pass can run ahead of the training step, which is how Mistral prefetches images and caches their features for its 99% of text-only MFU.

Moonshot dropped the SigLIP-initialised encoder it had used for K2.5: “When a pre-trained encoder is attached to the LLM, joint optimization becomes unstable: the SigLIP-initialized MoonViT-3D shows persistently higher gradient norms with frequent spikes”. It concludes that “contrastive pre-training is unnecessary as an initialization for multimodal language models at scale”. The evidence is gradient norms in ablations of unstated size, with an unquantified “matches” on vision evaluations, and the two encoders differ in architecture. Higher gradient norms are common in a pretrained part under a new loss (our reading), and the usual answer is a lower learning rate for it; the report doesn’t say what rate the SigLIP arm had. Every lab in the table that keeps a pretrained encoder slows it when it joins, by freezing it or a lower rate, except Xiaomi, whose V2.6 report states no such step. DeepSeek’s, K2.5’s and Xiaomi’s encoders also end their own training with a stage through a smaller LLM’s text loss (236B, about 1T, and over 4T image tokens) before joining. No source runs both initialisations under the same guard.

What the data is

Eight of the 12 labs name at least one type of multimodal training data (computed from the reports; Google, Thinking Machines, Z.ai and StepFun name modalities at most).

TypeLabs naming it (of 8)
Captions or alt-text pairs7: DeepSeek, Moonshot, Mistral (encoder stage), Qwen, Xiaomi, Liquid, NVIDIA
OCR6: DeepSeek, Moonshot, Mistral (encoder stage), Xiaomi, Liquid, NVIDIA
Interleaved image-text documents5: DeepSeek, Moonshot, Mistral, MiniMax, Liquid
Video5: Moonshot, MiniMax (as a modality), Qwen, Xiaomi, NVIDIA
Grounding or localisation4: DeepSeek, Moonshot, Xiaomi, NVIDIA
Code with its rendering4: DeepSeek, Moonshot, Xiaomi, NVIDIA
Charts and document question answering (VQA)3: DeepSeek (encoder stage), Liquid, NVIDIA
GUI or computer use3: DeepSeek, Xiaomi, NVIDIA
Audio-text or speech recognition4: Qwen, Xiaomi, Liquid, NVIDIA

Interleaved documents are web pages and PDFs kept with their images in place. DeepSeek’s account of building them is the most detailed. Its crawler was “overly biased toward text-centric web content”, so it “re-bootstrapped it from Common Crawl”, and because multimodal processing “incurs higher CPU and disk storage costs”, it filters in “progressively more expensive stages”, ending with quality scoring by a small vision-language model, SmolVLM. MiniMax “re-architected the entire text pretraining data pipeline” to produce interleaved data.

Grounding data ties a phrase to a place in the image, as a box or a point. Four labs name it: DeepSeek (“visual grounding and pointing”), Moonshot (“coordinate supervision … in both absolute and normalized ([0,1]) formats”), Xiaomi and NVIDIA. Liquid states the opposite for LFM2-VL’s SFT mixture, “we do not employ explicit grounding supervision such as bounding boxes or point coordinates” (no grounding supervision), and expects “weaker performance on fine-grained localization”. The other seven say nothing either way.

Found or generated

Labs split on whether multimodal data should be found or made, and some answer differently for pre-training and post-training. For its multimodal pre-training, DeepSeek “refrained from large-scale data synthesis”, on “the premise that raw web data naturally provides rich multimodal knowledge”; its post-training builds “large-scale, automated pipelines for data synthesis and environment construction” for agent tasks, without saying whether they make multimodal data. Mistral says “Almost all our datasets are natively multimodal”. MiniMax’s blog says “interleaved data scales more easily than synthetic data”, from “extensive experiments” it doesn’t show, which compares ease of scaling and doesn’t say MiniMax dropped synthetic data (D3).

Others make data on purpose. Moonshot combines open collections “with in-house pipelines for filtering, synthesis, and deduplication”, and for K3 says “we substantially scale up” programmatic multimodal data: code paired with its rendered output, in “SVG, 3D assets, Webpage, Game, and CAD schematics”. Rendering the code makes the picture, so the two match by construction (our reading). For K2.5 it set “strict limits on synthetic captions to mitigate hallucination”. Liquid recaptions open image datasets and generates question-answer pairs, as BLIP-2 (Li et al., 2023) did earlier with BLIP’s CapFilt method, which ranks ten generated captions and the original by image-text similarity and keeps the best two. NVIDIA generates SFT data with open models such as Qwen3.5 and Qwen3-VL, and long-document data “with NeMo Data Designer” for “documents spanning up to hundreds of pages”. Qwen trained its audio encoder on “audio-text pair data generated by Qwen3-ASR”, its own speech recogniser, so the audio is recorded and the text machine-made (our reading).

Images in post-training

In post-training, one lab measured what generated multimodal data did. NVIDIA used Kimi K2.6 and K3 “to generate agentic datasets with a focus on image tool operations such as crop, rotate, zoom, flip”, and K2.6 alone for video demonstrations in the SpatialClaw tool format, of which it kept “14,169 trajectories and 81,366 assistant actions”, each a recorded run of actions and tool results. In its Table 7, the mean over 20 spatial-reasoning and video benchmarks rises from 50.5 to 54.8 against a baseline that “uses no agentic vision training”, and all 20 rise, by 0.3 to 10.4 points (computed). The table doesn’t name the checkpoint or report variance, and the evaluation uses the same SpatialClaw tool format the data is written in, so part of the gain may be familiarity with the format.

Moonshot and Xiaomi put the tools in RL. K3’s visual-reasoning trajectories run “in an agent environment equipped with a Python interpreter in an isolated sandbox”, where the model writes code “to crop, zoom, or transform the input image” and gets the results back as observations (visual reasoning in a sandbox). MiMo-V2.6 trains visual agent tasks on websites, games, 3D scenes, slides and Figma designs, graded by rubrics and, for tasks that ask for a copy of a reference design, by “pixel-level similarity between rendered artifacts and their references”. With screenshots taken across turns, “a single trajectory can accumulate gigabytes of such data”, so rollouts ship only the change between requests.

Kimi K2.5 went the other way in SFT, in one preliminary comparison. SFT on text alone, with image operations written as code (“zero-vision SFT”), did better on visual agentic tasks than SFT with vision data: “text-vision SFT yields much worse performance on visual, agentic tasks, possibly because of the lack of high-quality vision data”. Its vision RL then raised text scores against the checkpoint before that stage: MMLU-Pro from 84.7 to 86.4, GPQA-Diamond from 84.3 to 86.4.

Audio, in brief

X2 covers how audio enters and in what order the omni models train. The hours labs report count different objects. Qwen trained its AuT audio encoder on 40 million hours of audio-text pairs and its speech-output model, the Talker, on “more than 20 million hours” of speech, and its headline is “more than 100 million hours of audio-visual data”. Xiaomi’s 20 million hours trained MiMo’s audio tokenizer. LFM2-Audio’s mid- and post-training mix holds “approximately 472,000 hours” of input and output audio per epoch, with the epoch count not stated. The three counts can’t be compared.

LFM2-Audio’s loss shows what “tokens” means for audio. It trains on text and on output audio codes, weighted by the count of each in a batch, and “We do not compute losses on continuous audio input embeddings”, so the 13% of its mid-training that is input audio is context, like an image.

Comparison

Entry pointWhat it keeps or gainsWhat is at riskProtection usedEvidence on text, with its baseline
First stepOne run trains both modalities togetherText tokens at a fixed budget; the modalities competing; a multimodal data pipeline at pre-training scaleEncoder frozen for some or all of pre-training (DeepSeek, Mistral); “careful dataset design”Mistral: no regret at equal text tokens, modest at equal total (perplexity, scale not stated); K2.5 ablation: early beats later entry at equal text tokens
Text stage, then mixedA text base trained on its ownText skill during the switchProjector warm-up (MiMo-V2.5, K2.5)K2.5 ablation (unstated size): text dips, then recovers, in the mid and late arms
After mid-trainingA mid-trained text modelText skill, a new connectorConnector stage, 100% to 30% ramp, encoder at 1/5 learning rateLFM2-VL: compared with other VLMs only
SFTA fully pre-trained text baseText and reasoning skillProjector warm-up, staged SFT, frozen embeddings in text RLNano Omni against its backbone: mixed, AIME25 −7.0, AA-LCR +5.1
Continued training of a finished modelA finished text (V4-Flash) or vision-language (Qwen3.5) modelText skillLocked LLM at first (Qwen3.5-Omni); not stated (Vision-Exp)Vision-Exp: “on par” with V4-Flash, no side-by-side numbers; Qwen3.5-Omni-Plus lower on 12 of 14 than Qwen3.5-Plus-Instruct

Where the labs sit

LabModelsEntry pointVolume or mix (kind)Encoder by stageData types namedGenerated multimodal dataMultimodal post-training
DeepSeekV4.1 Flash; Vision-ExpStep 0; continued training45T; 7:1 corpus tokens by sample kindOwn; frozen to 28T, then trained7Avoided in pre-training
MoonshotK3 (K2.5)Step 0 (K2.5: on a near-finished K2)(K2.5: 15T vision-text)From scratch, never frozen (K2.5: SigLIP init)6Programmatic; some synthesisSandbox RL; K2.5 zero-vision SFT
MistralLarge 4Step 0Own, 1.6B; frozen until SFT3Encoder trained in SFT
MiniMaxM3Step 02“scales more easily” than synthetic
Qwen3.5; 3.5-OmniPre-training; audio by continued pre-trainingOmni S2: 4.29T listed, ~4T stated3.5’s vision encoder; AuT from scratch3Machine-labelled audio textSpecialist teachers
XiaomiMiMo-V2.6Text stage, then omni26T + 22T; 27T + 3TOwn, pretrained; full model trained7Visual agent RL
LiquidLFM2-VL, LFM2-Audio (LFM3)After mid-training (VL)~50B; ~100B in three unitsSigLIP2; frozen, then 1/5 LR5Recaptions, generated VQA, audio chatsJoint SFT
NVIDIA3.5 Super VL (Nano Omni)SFTNano Omni: 466.9B SFT tokens, incl. prompts and repeatsC-RADIO; projector first, then trained8From Qwen, Kimi and other open models14,169 agentic trajectories; vision RL
GoogleGemma 4Pre-trainingFrozen in pre-training; none in 12B
Thinking MachinesInklingPre-training45T, share not statedNone (patch stem)Card: data “synthetically generated or augmented”, modality not statedImage and audio domains
Z.aiGLM-5.3-FlashPre-training30T corpus
StepFunStep 3.7 Flash, Step 51.8B encoder, training not stated

Data types count the rows of the table in “What the data is”. A blank means searched and not found. Step 3.7 Flash, Step 5, MiniMax M3.1 and the Qwen3.6 and Qwen3.8 cards give no multimodal training detail. LFM3’s vision, audio and omni models branch from a 22T-token base described in text terms only, so LFM2’s schedule shouldn’t be read onto them.

Easy to get wrong

  • “Native” as “from step 0”. Xiaomi, Liquid and Moonshot (for K2.5) use the word for recipes that train text first.

  • DeepSeek’s 7:1 as an image-token share. It divides corpus tokens between two kinds of sample, and the multimodal side includes its text; image tokens are fewer than one in eight, by an amount not stated.

  • The two 5:5:1s. MM1’s is caption:interleaved:text data; LFM2-VL’s is the learning rate of backbone, connector and encoder.

  • “On par” as no change. Qwen3.5-Omni-Plus is lower on 12 of 14 text benchmarks than the baseline it is called on par with.

  • “Enters at SFT” as little data. Nano Omni’s SFT stages hold 466.9B tokens, including a stage that trains the audio encoder.

  • The K2.5 ablation as the cost of vision to text. It has no text-only arm and holds text tokens equal in every arm.

Check yourself

  1. A lab wants its multimodal run to see as many text tokens as a 20T-token text-only run, and non-text tokens will be 20% of what it sees. How many tokens must the run see? If it stays at 20T, how many text tokens does it lose?

    Answer20T/(1−0.2)=25T20\text{T}/(1-0.2) = 25\text{T}, so 25% more. At 20T it sees 16T of text, 20% fewer. Mistral reports no regret in the first case and modest regret in the second, without saying its own share.
  2. LFM2-VL’s text-only fraction falls linearly from 100% to 30% over the first 5% of steps and then holds. What is the average text-only fraction over the first 10% of steps?

    AnswerThe first 5% average 65%, the next 5% sit at 30%, so (0.05×65%+0.05×30%)/0.10=47.5%(0.05 \times 65\% + 0.05 \times 30\%)/0.10 = 47.5\% (computed). The report doesn’t say whether the fraction counts samples or tokens.
  3. Nemotron 3.5 Super VL meets images only after pre-training, in a projector warm-up and then SFT. Name two protections NVIDIA’s recipes use for the finished text model, and say what the K2.5 ablation can and can’t tell you about that choice.

    AnswerA projector warm-up before SFT, and SFT in stages (two for Super, seven in Nano Omni, with the language model frozen during the audio stages and token embeddings frozen in text RL). K2.5 compares entry points inside pre-training at a fixed budget and an unstated size, so it says nothing about adding vision at SFT to a fully pre-trained base.

In the map

Methods

Taxonomy nodes

Further reading

↑