Open Models Almanac
Methodsmodel architecturemultimodal architecture

general family · filed under model architecture

Encoder-free multimodal architecture

An encoder-free model has no separate network that mixes image patches before the language model: each patch is mapped to an embedding on its own, by one linear layer (Gemma 4 12B) or a small per-patch MLP (Inkling), and the decoder does all mixing between patches. The other Gemma 4 sizes, including the 31B, use a 550M or 150M vision encoder.

Written for the explainer Getting images and video into a decoder: encoders, projectors, and native fusion.

Also called Encoder-free decoder-only architecture, encoder-free architecture, encoder-free multimodal input projection.

sources
6
models
2
labs adopt it
2
strongest
core

Documented in

Google DeepMind
Thinking Machines Lab

Evidence

6 quotes from 6 sources, one card per model, each opening with its strongest.

How sources treat it
used 2core 4

Gemma 4 31B

Google DeepMindadopted5 sources

Gemma 4 12B eliminates these encoders entirely, projecting raw image patches and audio waveforms directly into the LLM's embedding space through lightweight linear layers.

coremodel architecturein Gemma 4Google DeepMind
4 more from this model

Gemma 4 12B eliminates these encoders entirely, projecting raw image patches and audio waveforms directly into the LLM's embedding space through lightweight linear layers.

coremodel architecturein Gemma 4Google DeepMind

we trained Gemma 4 12B with an encoder-free architecture to integrate audio and vision input directly.

coremodel architecturein Gemma 4 12BGoogle

This unified approach means all modalities flow straight into a single decoder-only transformer

coremodel architecturein Gemma 4Google DeepMind

We introduce a unified encoder-free architecture for the 12B model, which projects raw 40ms audio chunks and image patches into the LLM embedding space, alleviating the need for separate encoders and reducing memory fragmentation.

usedmodel architecturein Gemma 4 12BGoogle DeepMind

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑