general family · filed under model architecture
Encoder-free multimodal architecture
Audio and vision inputs are incorporated without separate modality encoders, with raw inputs projected directly into the language-model embedding space.
Also called Encoder-free decoder-only architecture, encoder-free architecture, encoder-free multimodal input projection.
- sources
- 6
- models
- 2
- labs adopt it
- 2
- strongest
- core
How sources treat it
One count per evidence span, weakest treatment to strongest.
Documented in
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
Evidence
6 spans quoted from the sources, strongest treatment first.
This unified approach means all modalities flow straight into a single decoder-only transformer
we trained Gemma 4 12B with an encoder-free architecture to integrate audio and vision input directly.
Gemma 4 12B eliminates these encoders entirely, projecting raw image patches and audio waveforms directly into the LLM's embedding space through lightweight linear layers.
Gemma 4 12B eliminates these encoders entirely, projecting raw image patches and audio waveforms directly into the LLM's embedding space through lightweight linear layers.
We introduce a unified encoder-free architecture for the 12B model, which projects raw 40ms audio chunks and image patches into the LLM embedding space, alleviating the need for separate encoders and reducing memory fragmentation.
We opted for an encoder-free architecture for audio and vision inputs, consistent with the interaction model design.
Filed alongside
Other methods under model architecture :: multimodal architecture.