specific method · filed under model architecture
Discrete token encoding for audio
Audio is represented using discrete tokens rather than an unspecified continuous representation.
Also called discrete token encoding, discrete token encoding for audio inputs.
- sources
- 4
- model
- 1
- lab adopt it
- 1
- strongest
- core
How sources treat it
One count per evidence span, weakest treatment to strongest.
core 4
Documented in
Evidence
4 spans quoted from the sources, strongest treatment first.
audio via discrete token encoding
coremodel architecturein InklingThinking Machines Lab
audio via discrete token encoding
coremodel architecturein InklingThinking Machines Lab
audio via discrete token encoding
coremodel architecturein InklingThinking Machines Lab
audio via discrete token encoding
coremodel architecturein InklingThinking Machines Lab
Filed alongside
Other methods under model architecture :: multimodal architecture.
Encoder-free multimodal architectureEarly fusion multimodal trainingHierarchical patch encoderMixed-modality trainingConfigurable visual token budgetJoint multimodal decoding in a shared hidden spaceNative multimodal understanding3×3 pixel-unshuffle downsamplingJoint vision-language pre-trainingRaw audio projection into the LLM embedding spaceVision Transformer (ViT)2×2 pixel-shuffle downsamplingAudio encoder initialized from MiMo-AudioAudio Transformer (AuT)Causal streaming ConvNet codec decoderData-parallel-first multimodal encodingDecoupled Encoder Process (DEP)Dedicated multimodal encodersDisaggregated encoder trainingDiscarding the LLM after vision-encoder trainingDual-format coordinate supervisionExplicit text-string timestampsFour-frame audio patches with within-patch bidirectional self-attentionFrom-scratch vision-encoder training with next-token prediction