implementation detail · filed under model architecture
Raw audio projection into the LLM embedding space
Raw audio, including chunked audio, is projected directly into the language-model or text-token embedding space without an audio encoder.
Also called raw audio chunk projection.
- sources
- 2
- model
- 1
- lab adopt it
- 1
- strongest
- used
How sources treat it
One count per evidence span, weakest treatment to strongest.
used 2
Documented in
Evidence
2 spans quoted from the sources, strongest treatment first.
We removed the audio encoder entirely and projected the raw audio signal into the same dimensional space as text tokens.
usedmodel architecturein Gemma 4 12BGoogle
Raw audio is segmented into 40ms chunks at 16kHz, resulting in 640-dimensional vectors per chunk. These are projected directly into the LLM embedding space.
usedmodel architecturein Gemma 4 12BGoogle DeepMind
Filed alongside
Other methods under model architecture :: multimodal architecture.
Encoder-free multimodal architectureEarly fusion multimodal trainingDiscrete token encoding for audioHierarchical patch encoderMixed-modality trainingConfigurable visual token budgetJoint multimodal decoding in a shared hidden spaceNative multimodal understanding3×3 pixel-unshuffle downsamplingJoint vision-language pre-trainingVision Transformer (ViT)2×2 pixel-shuffle downsamplingAudio encoder initialized from MiMo-AudioAudio Transformer (AuT)Causal streaming ConvNet codec decoderData-parallel-first multimodal encodingDecoupled Encoder Process (DEP)Dedicated multimodal encodersDisaggregated encoder trainingDiscarding the LLM after vision-encoder trainingDual-format coordinate supervisionExplicit text-string timestampsFour-frame audio patches with within-patch bidirectional self-attentionFrom-scratch vision-encoder training with next-token prediction