A screenshot of 1024 by 1024 pixels takes 258 positions of context in Gemma 4 31B at its default setting and 2,705 in Inkling as vLLM, a common serving engine, runs it. Both counts are computed from the models’ own preprocessing code, as are all such counts below. The gap comes from how each model shrinks the image and groups its pieces, which is what this article is about. A 128K-token context holds about 500 such images in one model and about 48 in the other.
Every flagship in the map (the set of open-weight models this site tracks) that takes images turns them into vectors placed in the decoder’s input sequence, next to the text. The models differ in three choices: what turns pixels into those vectors (a separate image network of 0.4 to 1.8 billion parameters, or one to four layers applied to each patch alone), when that network and the language model learn together (from the first step of pre-training, or after the text model is finished), and how many vectors one image becomes.
Background: from pixels to patches
The image networks in this article are Vision Transformers (ViT; Dosovitskiy et al., 2020). A ViT cuts an image of pixels into square patches of pixels. When does not divide a side, the preprocessor resizes or pads the image first, so the patch count is
Each patch, numbers for RGB, goes through one linear layer to become a vector of width , and the vectors run through transformer layers with bidirectional attention. (, are pixel sizes; stays the series’ query heads, and , are kept apart from the backbone’s , .) The original ViT added a learned position table; most models here use 2D-RoPE, some alongside a table. RoPE rotates pairs of query and key dimensions by an angle proportional to the position (T4); the 2D version gives some pairs the row index and the rest the column. Patches are 14 or 16 pixels in every mapped model except Inkling (40) and Gemma 4 12B (48). At a 1024-pixel side is 73.1 patches, so DeepSeek, Kimi and GLM pad each side to the next multiple of 14 (of 28 where a 2×2 merge follows): 1,036 pixels, patches.
Each vector the language model receives is a position in the context window. It is processed in prefill, the compute-bound forward pass over the whole prompt. And its keys and values stay in the KV cache: during generation the model produces one token per step (decode) and, to avoid recomputing the prompt, stores every earlier position’s keys and values and reads them all at every step (00 has the cost model). The encoder runs once per image, in prefill, and stores nothing for decode, so most designs keep large inside the encoder and reduce it before the backbone. Position counts here include the delimiter tokens a processor adds (start, end, row breaks) unless called content tokens.
Encoder and projector
The standard design has three parts. A ViT encodes the patches. A merge step folds each 2×2 or 3×3 block of patch vectors into one, usually by concatenating them into a wider vector (variants below). A projector, usually a two-layer MLP, maps each merged vector to the language model’s embedding width. The tokenizer writes one placeholder id per visual position into the prompt, and the model overwrites those positions’ embeddings with the projector’s outputs (Kimi K3 expands a single placeholder instead). Once spliced in, an image position goes through the same decoder weights as text; no mapped model uses cross-attention for images.
Earlier designs used a fixed number of visual vectors: Flamingo (Alayrac et al., 2022) a Perceiver-style resampler of 64 learned queries that cross-attend to the patch features, BLIP-2 (Li et al., 2023) 32 such queries, its Q-Former. LLaVA (Liu et al., 2023) projected every patch feature instead, and LLaVA-1.5 made the projector a two-layer MLP; the mapped models all use this per-patch design.
For a 1024×1024 image, DeepSeek-V4.1-Flash’s 32-layer encoder sees 5,476 patches. Its 3×3 merge needs a grid divisible by 3, so the code pads the grid to with zeros and gets merged vectors, each wide; the projector maps them to 5,120, the backbone’s width. The processor adds a learned newline vector after each row and a start and end token: positions. Kimi K3’s 2×2 merge gives positions from the same 5,476 patches, with no delimiters (its image size goes into the prompt as text).
- padding (image border enlarged)
- visual positions
- row-newline vectors
- start / end tokens
- placeholder ids
- text (schematic)
DeepSeek-V4.1-Flash
- image 1024² → 1036² padded
- patches 74 × 74 = 5,476 of 14 px
- encoder 32 layers × 1,024, ≈ 412M · out 5,476 vectors
- merge 74 × 74 zero-padded to 75 × 75, 3×3 → 25 × 25 = 625 vectors of 9,216
- projector 9,216 → 5,120 → 5,120 · DeepseekV4Aligner
625 visual, 25 row-newline vectors (one after each row), 2 start and end tokens
Kimi K3
- image 1024² → 1036² padded
- patches 74 × 74 = 5,476 of 14 px
- encoder 27 layers × 1,024, ≈ 401M · out 5,476 vectors
- merge 2×2 → 37 × 37 = 1,369 vectors of 4,096
- projector 4,096 → 4,096 → 7,168 · PatchMergerMLPV2
no delimiter vectors: the image size goes into the prompt as text
Patch icons show a 6 × 6 corner of the grid; merge icons the bottom-right corner, where DeepSeek pads 74 to 75. Strip lengths are to scale against each other (text either side is schematic).
The parameters are small next to the backbone: Kimi K3’s encoder is 401M of 2.78T, about 0.014% (Kimi’s report), Step 3.7 Flash’s 1.8B of 198B (its card). Projectors range from 6.2M in Gemma 4 31B to about 240M in Step 3.7 Flash, counting its downsampling convolutions (computed). The compute is a larger share, because the encoder runs over every patch before the merge. Each weight does a multiply and an add per patch, and attention’s and each cost :
with the weights per layer and heads times head dimension (1,024 in DeepSeek-ViT, 1,536 in Kimi K3); patch embedding, norms and softmax left out. For a 1024×1024 image that is about 8.4 TFLOP for DeepSeek-ViT, 9.3 for Kimi K3 and 25 for Mistral Large 4’s 1.6B encoder (computed). Attention’s share is , about half for DeepSeek and Kimi here, and more beyond (about 6,300 patches for DeepSeek-ViT). At Kimi K3’s largest image, 3584×3584 pixels, it is about 760 TFLOP, 712 of it attention, as much as the prefill of a backbone with 23B active parameters over the image’s 16,384 positions.
The backbone does about its active parameters in matrix FLOPs per position (a mixture-of-experts model, MoE, has many FFNs per layer and runs a few per token; those are its active parameters). Dense Gemma 4 31B spends TFLOP on the image’s positions, and its encoder, on the image shrunk to 768×768, 3.2, a fifth of that. Qwen3.6-35B-A3B, with 3B active and the same encoder as the larger Qwen models, spends about 5 TFLOP on 1,026 positions and 5.5 in the encoder (computed, short prompt; about 2.4B of the 3B, leaving out the embedding and output tables, which do no per-position matrix work in prefill). The fewer active parameters, the larger the encoder’s share of prefill. In decode the encoder costs nothing; the image costs the KV bytes its positions left. The engineering cost is a second network whose work varies with image size and does not fit how the backbone is split across devices (systems section). Every mapped model with vision uses this design except Inkling and Gemma 4 12B, with encoders of 24 (GLM-5.3-Flash) to 47 (Step 3.7 Flash) layers.
Where the encoder comes from, and when it trains
A pretrained encoder was trained for another objective. SigLIP (Zhai et al., 2023), the usual starting point, trains an image and a text encoder so that matching image and caption embeddings score high. The loss sees one pooled embedding per image, so detail a caption does not mention, such as where each number sits in a chart, gets no direct signal; SigLIP 2 added losses on the un-pooled features partly for this. Kimi K3’s report puts it directly: a contrastive loss “favors global semantics over fine-grained textual and structural cues”. The labs answer it in three ways.
Some keep a pretrained encoder frozen, the classic transfer-learning setup with a trained projector stitched to the language model. Its limit is exact: the backbone sees the image only through the frozen features, so two images the encoder maps to nearly the same features stay indistinguishable however long the rest trains. LLaVA froze its encoder throughout, and SigLIP 2’s authors, citing PaliGemma, freeze it in a short Gemma 2 2B run, which “has essentially no impact on quality”; Qwen2-VL argued a “static, frozen CLIP-style” encoder is not enough and trained its ViT. Gemma 4’s models “have frozen vision and audio encoders”, and its audio section adds “As with the vision encoder, we keep weights frozen during pre-training” (later stages not stated). LFM2-VL trains only the connector (its word for the projector) first, then all three parts at learning rates of 5:5:1 for backbone, connector and encoder.
Others pretrain their own encoder, freeze it, and unfreeze it late. DeepSeek-ViT is pretrained in two stages of its own, SigLIP’s loss on about 47B image and alt-text pairs, then next-token prediction through a 4B MoE language model on 236B tokens, which is then discarded. In the 45T-token run it stays frozen “until the learning-rate decay stage”, which begins at 28T, and then trains at a smaller learning rate. Mistral Large 4 trains a 1.6B encoder “from scratch with a captioning objective” over 8T tokens and keeps it “frozen throughout pretraining and unfrozen only for supervised fine-tuning (SFT)”. MiMo-ViT is pretrained with a small language model on more than 4T image tokens, and the report then trains “the full model” jointly in its second stage.
Only Kimi K3 trains its encoder from random initialisation, jointly with the backbone: “we train Kimi K3 vision encoder, MoonViT-V2, entirely from scratch with next-token prediction”. The encoder has no loss of its own; with the usual loss mask no target sits at an image position, so its gradient arrives only through the backbone’s mixing layers (KDA and attention in K3) from the text after the image, and an image with no text after it teaches it nothing. The stated reason is stability: the SigLIP-initialised MoonViT-3D “shows persistently higher gradient norms with frequent spikes, while MoonViT-V2 remains stable”. The from-scratch encoder “matches the SigLIP-initialized baseline across vision evaluations”, in ablations of unstated size, with no numbers. (The two encoder-free stems below are also trained from scratch.) A frozen encoder, in turn, needs no backward pass and can run ahead of the step: Mistral prefetches and caches its features, “sustaining 99% of text-only MFU” (model FLOPs utilisation, the share of peak arithmetic achieved). Labs that train the encoder recover most of this by scheduling.
| Model | Encoder origin | Encoder trained when |
|---|---|---|
| Kimi K3 | random init, next-token loss through the backbone | jointly from step 0 |
| DeepSeek-V4.1-Flash | own; SigLIP loss, then next-token prediction through a 4B LLM | frozen until 28T of 45T, then trained |
| Mistral Large 4 | own, 1.6B; captioning objective, 8T tokens | frozen in pre-training, trained in SFT |
| MiMo-V2.6 | own, 681M; trained with a small LLM, >4T image tokens | full model trained in the second pre-training stage |
| Gemma 4 (not 12B) | 550M (150M for the small E2B/E4B); origin not stated | frozen in pre-training; later not stated |
| LFM2-VL | SigLIP2 (86M or 400M) | frozen for connector training, then at 1/5 the LR |
| Nemotron 3.5 Super VL | C-RADIO (NVIDIA’s; pretrained elsewhere, our reading) | “vision components” trained in SFT; frozen in its video teacher’s RL (final RL not stated) |
| Qwen3.5, MiniMax-M3, Step 3.7, GLM-5.3 | not stated | not stated |
Sources: the labs’ reports and cards; “not stated” means searched and not found.
When vision joins the backbone
The language model can meet images from its first training step, or after it has learned text, and the sources call both “early fusion” or “native”, as well as a third thing. Chameleon (2024) used “early fusion” for a form of input: each 512×512 image became 1,024 discrete codes from an 8,192-entry codebook in the text vocabulary, and one transformer trained on the mixed sequence from scratch, generating images as well as reading them. No mapped model does this; all feed continuous vectors and compute their loss on text. The mapped labs mean timing, image data in the pre-training mix from the start, which this article calls “fused from the start”. Chameleon’s 7B diverged “after approximately 20% of a training epoch” without QK-Norm (layer norm on queries and keys), which the paper ties to generating images (“ablations without image generation did not diverge”).
- text-only data
- mixed data, encoder frozen
- mixed data, encoder trained
- mixed, encoder training not stated
- SFT (dashed: encoder not stated)
- MiMo-V2.6-Flash26T text · 22T omni
- MiMo-V2.6-Pro27T text · 3T omni
- DeepSeek-V4.1-Flashmixed, encoder frozen to 28T · 17T, encoder trained
No stated token counts: not to scale, in the lab's words
- Kimi K3joint from the start
- MiniMax-M3mixed “from Step 0”
- Mistral Large 4mixed from scratch, encoder frozen · SFT
- Nemotron 3.5 Super VLtext pre-training · vision data from SFT, after a projector warm-up; encoder training not stated
- LFM2-VLlanguage pre- and mid-training · ≈ 50B joint
Hover, focus or tap a row for the source's words.
Four labs say they fuse from the start: Kimi K3 (“language and vision are jointly optimized from the start of training”), MiniMax-M3 (“mixed-modality training from Step 0”), DeepSeek-V4.1-Flash (“multimodal data incorporated from the start of language-model pre-training”, at a 7:1 ratio of text-only to multimodal tokens) and Mistral Large 4 (“We train from scratch on interleaved image–text data”). DeepSeek and Mistral keep the encoder frozen for most or all of pre-training, so fusing from the start and training the encoder from the start are separate choices. Qwen3.5 reports “Early fusion training on trillions of multimodal tokens” without saying when. Gemma 4, Inkling (45T tokens “of text, images, audio and video”) and GLM-5.3-Flash (“30T-token multimodal pre-training corpus”) put images in pre-training without giving the step at which they enter; Step 3.7 Flash says nothing.
Others train text first. MiMo-V2.6-Flash trains “the language backbone on text-only data” for 26T tokens, then 22T on mixed data (MiMo-V2.5 adds a projector warm-up between). Nemotron 3.5 Super VL’s base “is pre-trained on text-only data” and meets vision data in SFT; LFM2-VL and LFM3-VL branch off finished text models; DeepSeek-V4-Flash-Vision-Exp added “visual modules” by continued training. “Native” does not separate the groups: MiMo-V2.5 calls itself “a native omnimodal model”.
Mistral gives the only statement about what images cost text, in perplexity and at an unstated scale: “we observe no regret in text perplexity when matching the number of text tokens seen; while matching the total training horizon yields modest regret”. That is, image tokens did not hurt text when added on top of the text budget, and cost something when they replaced text tokens in a fixed one. Qwen3.5’s “cross-generational parity with Qwen3” compares generations, not recipes. On the retrofit side, DeepSeek’s release note calls V4-Flash-Vision-Exp “on par with the official DeepSeek-V4-Flash” on pure-text tasks. Nemotron gives the late-fusion difficulty in a sentence: two SFT stages “because learning vision and text capabilities simultaneously is challenging”. No lab reports a comparison of fusion timing on the model it shipped, and a lab that tried the other route and dropped it would have little reason to publish that.
Joint training also affects the MoE router (C3). DeepSeek keeps experts evenly used with a per-expert
bias on the router’s scores, raised for underused experts and lowered for busy ones after each step, not by
gradient. Because image and text tokens “may induce different expert-routing preferences”, it keeps
“separate expert-wise correction biases for text and image tokens” (in the code, a second bias bias_vl on
every MoE layer). No other lab describes modality-specific balancing.
Encoder-free: a patch stem instead of an encoder
If vision is trained into the backbone anyway, the encoder could go. Two mapped models say they have done this, and both still have a small vision-specific layer. The line used here is the receptive field: an encoder’s output at one position depends on other patches, through attention or convolutions that span several patches; a stem’s output depends on one patch only. Under this test the 24-to-47-layer ViTs above are encoders and the two “encoder-free” models have stems.
Gemma 4 12B “replaces the 550M vision encoder by a single large matmul (35M parameters)” on 48×48×3 pixel patches, plus “2D coordinate-based positional embeddings” and a LayerNorm, “alleviating the need for separate encoders and reducing memory fragmentation”. The other sizes keep their encoders; no 12B config is cached, so this is the report’s account.
Inkling encodes images “as patches of 40x40 pixels using a four-layer hMLP”, the hierarchical MLP stem
of Touvron et al. (2022), and its blog calls the design “encoder-free”; its model card’s “hierarchical
patch encoder” is the same module. In the code (HMLPPatchEncoder) the four layers fold a patch in
stages, 5×5 pixel blocks, then 10×10, then 40×40, then two frames (a still image goes in as two copies,
see Video), with no attention and nothing crossing a patch boundary: a small convolutional network
whose receptive field is one patch, about 84M parameters (computed). Each 40×40 patch becomes one
backbone position.
- pixels per position
- 1,024 (32²)
- vision parameters
- ≈ 415M encoder + 40M projector
- pixels per position
- 2,304 (48²)
- vision parameters
- 35M (report)
- pixels per position
- 1,600 (40²; × 2 frames in the code)
- vision parameters
- 83.7M (four linears)
Patch sizes are drawn to one pixel scale. Only the encoder has lines between patches: in the two stems, patches first meet in the decoder's own attention. Gemma 4 12B's numbers are from its report; no config was checked.
Without an encoder, a merge is just a bigger patch, so the patch size sets the token count, and all mixing between patches happens in the backbone’s layers rather than a 0.4B encoder’s. Gemma 4 12B’s one matmul on 6,912 inputs can represent the 31B’s 16-pixel patch layer, 3×3 average and projector composed into one linear map once the 27 encoder layers between them are removed (our reading; the 12B’s weights are unconstrained, the composed map has nine tied blocks); its 48-pixel patches land on the same grid as the 31B’s pooled one, so at each budget the counts would match (our inference from the 31B’s resize rule). Inkling’s 40-pixel patches give 677 positions for a 1024×1024 image at native size, within the encoder models’ range, but its evaluations and vLLM double the long edge first, which gives 2,705, the most in the map. That suggests, as our reading, that a stem needs more positions to see fine detail.
The backbone also inherits two jobs an encoder did. Its attention is causal, so a patch cannot see patches later in reading order unless image positions attend both ways, as in Gemma 4’s sliding-window layers (whether Inkling does is not stated). And layout must come from somewhere: Gemma 4 12B adds 2D coordinate embeddings, while Inkling’s stem adds no positions or row breaks, leaving raster order.
Published encoder-free models report slower training. Fuyu-8B published no training account. EVE (Diao et al., 2024), retrofitting vision into the 7B text model Vicuna, found that “direct training without encoders results in slow convergence and large performance gaps”, and closed part of the gap by distilling a CLIP encoder into the decoder: an MSE loss pulls the decoder’s hidden states at image positions towards CLIP’s features in training. Its patch layer also mixes patches locally, and it is a retrofit, not a model trained from scratch like Gemma 4 12B.
How many tokens an image costs
The merge step is where most of the reduction happens, and most models use one operator under several names. Space-to-depth takes each block of patch vectors of width and concatenates them into one vector of width , which a linear layer then maps to width . Together the two are exactly a convolution over the patch grid with kernel and stride :
Kimi K3, Qwen3.5, MiMo, Nemotron, DeepSeek and Mistral Large 4 write it as concatenation then a linear layer or MLP, GLM-5.3-Flash as a strided convolution; they differ in (2, or 3 for DeepSeek) and . The labs call the concatenation “pixel shuffle” or “pixel unshuffle”; in Shi et al. (2016) the two are inverses, and every model here applies the downsampling one. MiniMax-M3 projects each patch to 6,144 before merging, hence its 234M projector. Step 3.7 Flash uses two overlapping convolutions (kernel 3, stride 2), ÷16. Gemma 4 averages each 3×3 block, the case where all nine taps are equal and fixed, keeping width 1,152; the arrangement inside a block survives only as far as the encoder wrote it into the features.
The other lever is the image size the preprocessor allows, and here the models differ more. Gemma 4
lets the caller pick a budget of 70, 140, 280, 560 or 1,120 tokens. At 280 it scales the image so that
its 16-pixel patches, pooled 3×3, fit, and floors each side to a multiple of 48 pixels: 1024 becomes 768,
patches pooled to tokens, plus a start and an end token, 258. The
five budgets give 64, 121, 256, 529 or 1,089 tokens for our image; “A higher token budget preserves more
visual detail at the cost of additional compute”, says its card. DeepSeek-V4.1-Flash caps an image at
max_image_tokens: 1024 and pads to keep its aspect ratio. Qwen3.5’s field longest_edge holds a pixel
count, 16,777,216 for images, so our image passes at full size: pixels per position after
the 2×2 merge, 1,024 positions plus two. MiniMax-M3 shrinks any image larger in area than 672×672 pixels,
and gives 576 positions plus two.
These are processor defaults, not always the labs’ evaluation settings: Gemma 4’s main vision table uses “the maximal supported resolution (1120 vision tokens)” (280 in an appendix table), and MiniMax’s use images up to “long edge max = 3584 pixels”. Large images are tiled or kept at native resolution. Tiling cuts crops of the encoder’s training size plus a downscaled thumbnail: LFM2-VL uses 2 to 10 tiles of 512×512 (up to about 2,800 tokens; tiling off in most of its mid-training runs), Step 3.7 Flash a 728×728 whole view, squashed whatever the aspect ratio, plus 504×504 crops. Native resolution, from NaViT and Pixtral, keeps the image’s shape within a pixel budget, as most mapped models do. None of these rules is learned: the model trains on what its processor produced, so a serving path that resizes differently feeds it inputs it may not have seen.
1024×1024 image (width × height): 258 to 2,705 positions.
- image content positions
- start / end, row-break and newline tokens
A larger downsampling factor discards detail the task may need; LFM2-VL states the loss for its ÷4 step as “only a slight degradation in downstream multimodal performance”. Mistral Large 4 adds a break token after each row (the last is the end token), 37 for our image, and DeepSeek a newline vector per row, so the backbone knows where rows end. Positions are not bytes: a position costs a full KV entry in a standard attention layer and much less in hybrid or compressed backbones such as Nemotron’s or DeepSeek-V4’s (T2, T3, T5).
Inkling’s count depends on the serving code. Its Hugging Face processor encodes a 1024×1024 image at full size, 677 positions; vLLM’s path doubles the long edge to 2,048 before patching, 2,705 positions, four times as many for the same model and image (computed from both code paths). The lab’s model card states the same doubling for its evaluations.
Video
Most models pair video frames into tubelets: the patch layer is a 3D convolution over two consecutive frames, so two frames cost one frame’s positions. The idea comes from video models (ViViT, Arnab et al., 2021); Qwen2-VL brought it to vision-language models, feeding a still image “as two identical frames” so one layer serves both. Qwen3.5, MiniMax-M3, Nemotron 3.5 Super VL, MiMo and GLM-5.3-Flash have a temporal patch of 2. Gemma 4 encodes frames one at a time at a budget of 70 tokens, each after a timestamp in text.
The frame rate then determines the count: 2 per second by default in Qwen3.5 and GLM-5.3-Flash, 1 in
MiniMax-M3 when its processor’s frame sampling is switched on (it is off by default). Gemma 4’s processor
samples 32 frames whatever the length; its card supports up to 60 seconds at one frame per second, which
at the video budget of 70 (64 tokens a frame) is 3,840 tokens for a minute. Qwen3.5 caps a clip’s total pixels, which shrinks frames as clips get
longer; its card suggests raising longest_edge to 469,762,048 pixels, “corresponding to 224k video
tokens”, for hour-long videos.
Nemotron 3.5 Super VL is the one mapped model that drops video positions inside the model, after the
encoder. Efficient video sampling (EVS) keeps a fixed share, the positions that differ most from the same
place in the previous tubelet: 30% at the config’s video_pruning_rate: 0.7, 50% at the --video-pruning-rate 0.5 of its card’s vLLM command; the first tubelet is always kept. The
share does not depend on content; a static scene loses less information, not more positions. The rule has
no parameters and runs at inference; no source says whether training saw pruned video. GLM-5.3-Flash marks
each tubelet with the image placeholder token between video delimiters (“the same image token is reused for
video frames”, its vLLM recipe), which concerns the token id, not shared features.
Kimi K3’s report says “attention is factorized into intra-frame spatial and inter-frame temporal passes”; the published code instead lets patches from up to four frames attend jointly and averages over them in the merge, and its published processor accepts only images, so the shipped video path cannot be checked.
- positions at the processor's defaults (Nemotron: the card's serving setting)
- before pruning
- upper bound
At its defaults GLM-5.3-Flash turns one minute into 82,140 positions and allows up to 240,000 per clip; the cheapest settings sample time coarsely.
Positions, briefly
Inside the encoder, the mapped models use 2D-RoPE alone (DeepSeek-ViT, Mistral Large 4, MiMo,
GLM-5.3-Flash), 3D-RoPE over time, row and column (MiniMax-M3), an interpolated learned table (Nemotron’s
RADIO), or both (Gemma 4, Kimi K3, Qwen3.5, Step 3.7). Inside the backbone, most give visual positions
ordinary 1D positions in reading order; Kimi K3’s backbone uses “no explicit positional embedding (NoPE)”. Qwen3.5 is the one whose cached code uses multimodal RoPE (M-RoPE, "mrope_section": [11, 11, 10], rotary frequency pairs split among time, row and column), which gives an image position its row
and column inside the backbone too; GLM-5.3-Flash’s backbone code is not cached. Three models also tell
the backbone where rows end, with row-break tokens (Mistral, DeepSeek) or the image size as text (Kimi
K3, image 1024x1024); the others rely on the encoder having written position into each vector, which
a stem cannot do. M-RoPE and audio-video time alignment are covered in X2, position encoding in general in
T4.
The encoder in the training and serving stack
In training, labs run the encoder as its own phase. Kimi’s “Decoupled Encoder Process”, from Kimi K2.5, “splits ViT and text training into separate stages and balances vision forward and backward passes across PP stages”. Pipeline parallelism (PP) splits the backbone’s layers across devices in sequence and leaves devices idle while the first and last micro-batches pass through (the “bubble”); K3 moves most encoder passes into those idle slots, and splits a single large image’s patches across devices, which gather each other’s keys and values for attention. DeepSeek runs encoder forward, language model forward and backward, then encoder backward, so the middle phase “preserves the parallel strategy of text-only training”. MiMo-V2.6’s RL training copies the encoder to every device of a tensor-parallel group (which splits each layer’s matrices across devices) and balances images across them. Images of different sizes are packed into one encoder sequence with attention kept inside each image, as in NaViT, so attention costs per image.
In serving, the encoder is small enough to copy: vLLM’s --mm-encoder-tp-mode data (its Qwen3.5 recipe)
gives each device a full encoder and splits the images, using “additional memory”. Repeated images can skip
work: an engine can cache an image’s encoder output, and prefix caching can reuse its positions’ KV, which
matters for agents that resend screenshots every turn (MiMo-V2.6’s RL rollouts: “their visual tokens already
sit inside the cached KV”). The image budget also reserves memory, since an engine must leave room for the
largest image it accepts; with --language-model-only, Qwen3.5’s card says, vLLM “skips the vision encoder
and multimodal profiling to free up memory for additional KV cache”. Encoders usually stay in BF16 under a quantized backbone (Nemotron keeps “C-RADIO encoder” and
“VL connector” in BF16 in every recipe). 02 covers the rest.
Comparison
| Design | Positions, 1024×1024 | Trained when | Main cost | Models |
|---|---|---|---|---|
| Frozen pretrained encoder + projector | 258 (Gemma 4 31B) | encoder before; projector with the LM | features fixed by another objective | Gemma 4; LFM2-VL for connector training |
| Own encoder, unfrozen late | 652–1,406 | encoder pretrained, frozen, then trained | separate encoder pipeline | DeepSeek-V4.1, Mistral Large 4 |
| Encoder trained jointly from scratch | 1,369 | with the backbone from step 0 | encoder backward every step | Kimi K3 |
| Text first, vision added | 1,026 (MiMo, Nemotron) | after text pre-training | vision arrives after text capacity is set | MiMo-V2.5/V2.6, Nemotron 3.5 Super VL, LFM2-VL, LFM3-VL |
| Encoder-free stem (one matmul or a per-patch MLP) | 677–2,705 (Inkling); 258 at 280 (Gemma 4 12B, inferred) | with the backbone, from scratch | backbone does all mixing; slower training (EVE) | Gemma 4 12B, Inkling |
| Discrete-token early fusion | 1,024 per 512² (Chameleon) | from scratch | tokenizer weak on text in images; instability when generating images | none in the map |
Position counts include delimiters and are computed from processor code (Gemma 4 12B inferred); “trained when” is from the labs’ reports.
Where the models in the map sit
Computed from the cached configs, processor configs and modeling code at pinned revisions, for one 1024×1024 image at default settings; encoder parameters are our estimates unless a lab states them.
| Model | Encoder (layers × width) | Params | Patch | Merge | Projector | Positions | Video |
|---|---|---|---|---|---|---|---|
| DeepSeek-V4.1-Flash | DeepSeek-ViT 32 × 1024 | 412M | 14 | 3×3 concat | 2-layer MLP, 73M | 652 | none |
| Gemma 4 31B | 27 × 1152 | 550M (report) | 16 | 3×3 average | 1 linear, 6M | 258 | frames one at a time |
| Kimi K3 | 27 × 1024 | 401M (report) | 14 | 2×2 concat | 2-layer MLP, 46M | 1,369 | ≤4-frame chunks (code); no video processor |
| Mistral Large 4 | Pixtral-style 30 × 2048 | 1.6B (report) | 14 | 2×2 concat + linear | 2-layer MLP, 67M with merge | 1,406 | none |
| MiniMax-M3 | 32 × 1280 | 631M | 14 | 2×2 concat, after projecting | 4 linears, 234M | 578 | tubelets of 2, 1 fps |
| Nemotron 3.5 Super VL | C-RADIO, 32 × 1280 | 653M | 16 | 2×2 concat | 2-layer MLP, 189M | 1,026 | tubelets of 2, EVS |
| Qwen3.5 / 3.6 / 3.8 | 27 × 1152 | 415M | 16 | 2×2 concat | 2-layer MLP, 31–40M | 1,026 | tubelets of 2, 2 fps |
| Step 3.7 Flash / Step 5 | 47 × 1536 | 1.8B (card) | 14 | two overlapping stride-2 convs, ÷16 | 1 linear (+ convs) | 504 | none |
| Inkling | none (4-layer per-patch MLP) | 84M | 40 | none | last stem layer | 677 (HF) / 2,705 (vLLM) | not in published code |
| MiMo-V2.5 / V2.6 | 28 × 1280, 24 window + 4 global | 681M (V2.6 report) | 16 | 2×2 concat | 2-layer MLP, 47–58M | 1,026 | tubelets of 2 |
| GLM-5.3-Flash | 24 × 1024 | 404M | 14 | 2×2 strided conv | conv + SwiGLU MLP, 159M | 1,371 | tubelets of 2, 2 fps, image token id |
Eighteen cached configs carry a vision_config (revisions counted separately), plus
DeepSeek-V4-Flash-Vision-Exp with the same encoder under flat fields and a 384-token budget. MiMo’s 24
windowed layers address the term at high resolution (its V2.5 card gives 729M for the same layout).
Easy to get wrong
- “Native” and “early fusion” do not say when the encoder trained (DeepSeek-V4.1: images from step 0, encoder frozen until 28T of 45T), and Chameleon’s “early fusion” means discrete image tokens, which no mapped model uses.
- “Pixel shuffle” and “pixel unshuffle” name inverse operations in general, but in these models both mean space-to-depth followed by a linear layer, a strided convolution. Gemma 4’s 3×3 step is average pooling, the fixed-weight case with the same ÷9.
- Encoder-free models still have vision-specific weights (35M in Gemma 4 12B, 84M in Inkling), and at native size they use as many positions per image as encoder models. Inkling’s evaluations and vLLM upscale first, which makes it the most expensive in the map.
- The encoder’s share of parameters (0.014% in Kimi K3) says little about its share of prefill: about 3 to 25 TFLOP per 1024×1024 image in the encoders computed here, close to the backbone’s own cost for a model with 3B active parameters, and growing with at large images. It costs nothing in decode.
longest_edgein Qwen’s processor configs is a pixel count. Read as an edge length, 469,762,048 is off by a square root.- Token counts belong to a processor and its settings: Gemma 4’s main vision table uses 1,120 tokens, not the default 280, and Inkling’s 677 under Hugging Face code is 2,705 under vLLM.
Check yourself
- Qwen3.5 uses 16-pixel patches and a 2×2 merge, wraps each image in a start and an end token, and lets a
768×1024 image through at full size. How many backbone positions does the image take?
Answer
One position covers pixels, so the grid is content positions, plus the start and end tokens: 770. In the calculator the solid segment shows 768 and the light end segment the two delimiters. - DeepSeek-V4.1-Flash and Kimi K3 both see 5,476 patches for a 1024×1024 image. Why does one hand 625
vectors to its backbone (before delimiters) and the other 1,369, and what does DeepSeek’s choice cost in
its projector?
Answer
DeepSeek merges 3×3 blocks; 74 is not divisible by 3, so the grid is padded to 75 and gives . Kimi merges 2×2: . DeepSeek’s merged vector is wide and its projector maps it to the backbone’s 5,120, so the first layer has M weights, and each backbone position summarises nine patches instead of four. - Mistral Large 4 freezes its encoder during pre-training and prefetches its features ahead of each step.
Why can Kimi K3 not do the same, and how does it keep the encoder’s cost down instead?
Answer
Kimi K3’s encoder trains with the backbone from step 0, so its outputs depend on the current weights and need a backward pass; they cannot be computed ahead on an old copy. Kimi schedules the encoder’s forward and backward passes into the pipeline’s idle slots instead. - Gemma 4 at its 280 budget turns a 1024×1024 image into 258 positions, at 1,120 into .
How many such images fit in a 128K context (131,072 positions) beside 8,000 tokens of text?
Answer
images at 280, and at 1,120.
In the map
Methods
- Native multimodal architecture 5 sources
- Encoder-free multimodal architecture 6 sources
- Early-fusion multimodal training 5 sources
- Vision Transformer (ViT) 2 sources
- Configurable visual token budget 3 sources
- Hierarchical patch encoder 4 sources
- Mixed-modality training from the start 4 sources
- Joint vision-language pre-training from the start 3 sources
- 3×3 PixelUnshuffle downsampling 2 sources
- C-RADIO vision encoder 2 sources
- 2D rotary position embeddings 2 sources
- Modality-specific load balancing 1 source
- Decoupled Encoder Process (DEP) 1 source
- Data-parallel vision encoding 1 source
Taxonomy nodes
- multimodal architecture 154 methods
Further reading
- Dosovitskiy et al., An Image is Worth 16x16 Words (2020), arXiv:2010.11929: the patch embedding, , position-table interpolation, and why ViTs need large training sets.
- Alayrac et al., Flamingo (2022), arXiv:2204.14198, and Li et al., BLIP-2 (2023), arXiv:2301.12597: the fixed-size connectors (a resampler with cross-attention, the Q-Former) that the per-patch projector replaced.
- Liu et al., Visual Instruction Tuning (2023), arXiv:2304.08485, and Improved Baselines (2023), arXiv:2310.03744: the per-patch projector and the two-stage freeze recipe.
- Zhai et al., SigLIP (2023), arXiv:2303.15343, and Tschannen et al., SigLIP 2 (2025), arXiv:2502.14786: what a pretrained encoder is trained to do, and the losses added for local detail.
- Dehghani et al., NaViT (2023), arXiv:2307.06304, and Agrawal et al., Pixtral 12B (2024), arXiv:2410.07073: images at their own size and shape, packed into one sequence.
- Wang et al., Qwen2-VL (2024), arXiv:2409.12191: dynamic resolution with a 2×2 merge, M-RoPE, and video as two-frame tubelets.
- Chen et al., InternVL 1.5 (2024), arXiv:2404.16821: pixel shuffle and aspect-ratio tiling with a thumbnail.
- Chameleon Team, Chameleon (2024), arXiv:2405.09818: discrete-token early fusion from scratch, and the stability fixes it needed.
- Diao et al., Unveiling Encoder-Free Vision-Language Models (EVE, 2024), arXiv:2406.11832: what training without an encoder takes, including a CLIP teacher distilled into the decoder.
- The Gemma 4 (arXiv:2607.02770, §2.1–2.3), Kimi K3 (§2.4, §5.2) and DeepSeek-V4.1 (§2.1, §2.5, §4) reports: the most detail on encoder, token budget and training schedule among the mapped labs.
