Open Models Almanac
Methodsmodel architecturemultimodal architecture

methods · model architecture

multimodal architecture

154 methods filed at this node or below it, from the sources of 32 models.

In this branch 154

Everything filed at this node or below it, with one collapsible heading per child node.

filed here 154

Encoder-free multimodal architecture core · 6 sources · 6 quotes
Early-fusion multimodal training core · 5 sources · 5 quotes
Native multimodal architecture core · 5 sources · 5 quotes
Discrete audio token encoding core · 4 sources · 4 quotes
Hierarchical patch encoder core · 4 sources · 4 quotes
Mixed-modality training from the start core · 4 sources · 4 quotes
Separate flow-matching action expert core · 3 sources · 3 quotes
C-RADIO vision encoder core · 2 sources · 2 quotes
Multimodal input core · 2 sources · 2 quotes
Native omnimodal architecture core · 2 sources · 2 quotes
Per-layer key-value conditioning core · 2 sources · 5 quotes
Reduced action-expert hidden size core · 2 sources · 2 quotes
Small pretrained VLM backbone core · 2 sources · 2 quotes
Visual token reduction core · 2 sources · 2 quotes
3×3 PixelUnshuffle downsampling used · 2 sources · 2 quotes
Raw audio projection into LLM embedding space used · 2 sources · 2 quotes
Vision Transformer (ViT) used · 2 sources · 2 quotes
Configurable visual token budget optional · 3 sources · 3 quotes
134 single mentions
Continuous action representation core · 1 source · 1 quote
Continuous DiT-style action expert core · 1 source · 1 quote
Convolution-attention vision encoder core · 1 source · 1 quote
ConvStack core · 1 source · 1 quote
Cross-embodiment modeling core · 1 source · 1 quote
Depth-aware perception tokens core · 1 source · 2 quotes
Dual proprioception conditioning core · 1 source · 1 quote
Dual-system compositional architecture core · 1 source · 1 quote
Eight-codebook Mimi RVQ codes core · 1 source · 1 quote
Embodiment-indexed proprioception MLP core · 1 source · 1 quote
Embodiment-specific action heads core · 1 source · 1 quote
Fused SigLIP-DINOv2 vision encoder core · 1 source · 1 quote
Hierarchical convolutional decoder core · 1 source · 1 quote
Intermediate VLM-layer features core · 1 source · 1 quote
Joint multimodal input understanding core · 1 source · 1 quote
Language backbone core · 1 source · 1 quote
Last-four-layer action-head conditioning core · 1 source · 1 quote
Learnable image queries core · 1 source · 1 quote
Multi-layer ViT feature concatenation core · 1 source · 1 quote
Multi-scale image cropping core · 1 source · 1 quote
Multi-view image input core · 1 source · 1 quote
Multimodal input understanding core · 1 source · 1 quote
Native visual and audio understanding core · 1 source · 1 quote
Pairwise sampled-frame embedding core · 1 source · 1 quote
Per-embodiment action MLPs core · 1 source · 1 quote
Perceptual-query token appending core · 1 source · 1 quote
Pixtral vision encoder core · 1 source · 1 quote
Robot-specific input and output layers core · 1 source · 1 quote
Separate RGB and depth patch embeddings core · 1 source · 1 quote
Shared action trunk core · 1 source · 1 quote
SigLIP2 pretrained vision transformer core · 1 source · 1 quote
Single resized crop per image core · 1 source · 2 quotes
Spatial patch-merge projector core · 1 source · 1 quote
T5 pretrained text encoder core · 1 source · 1 quote
Thinker–Talker architecture core · 1 source · 1 quote
Three-stage multimodal training protocol core · 1 source · 1 quote
Time-conditioned DiT modulation core · 1 source · 1 quote
Vision encoder core · 1 source · 1 quote
Vision encoder and multimodal aligner core · 1 source · 1 quote
Vision-language connector core · 1 source · 1 quote
Vision-language model backbone core · 1 source · 1 quote
Global-image-only runtime input default · 1 source · 1 quote
ASR-based connector alignment used · 1 source · 1 quote
Audio Transformer (AuT) used · 1 source · 1 quote
Causal streaming ConvNet codec decoder used · 1 source · 1 quote
Cross-attention-based resampler used · 1 source · 2 quotes
Data-parallel-first multimodal encoding used · 1 source · 1 quote
Decoupled Encoder Process (DEP) used · 1 source · 1 quote
Domain-specific resamplers and action heads used · 1 source · 2 quotes
Dual-format coordinate supervision used · 1 source · 1 quote
Dynamic image tiling used · 1 source · 1 quote
End-to-end autonomous driving used · 1 source · 1 quote
Explicit text-string timestamps used · 1 source · 1 quote
FiLM used · 1 source · 1 quote
Frozen encoders during pre-training used · 1 source · 1 quote
Grid quantization of depth maps used · 1 source · 1 quote
Index-token marking for multi-image inputs used · 1 source · 1 quote
Joint full-model multimodal training used · 1 source · 1 quote
Learned image-padding-status embeddings used · 1 source · 1 quote
Lightweight vision embedding module used · 1 source · 1 quote
Masking missing robot image slots used · 1 source · 1 quote
MiMo-Audio-initialized audio encoder used · 1 source · 1 quote
Molmo2-ER backbone used · 1 source · 1 quote
MoonViT-V2 used · 1 source · 1 quote
Multi-resolution visual feature fusion used · 1 source · 1 quote
Parallel action prediction used · 1 source · 1 quote
PixelUnshuffle visual-token downsampling used · 1 source · 1 quote
Pre-trained vision and language encoders used · 1 source · 1 quote
Pretrained vision-language encoder used · 1 source · 1 quote
Projector warmup used · 1 source · 1 quote
Repurposed navigation vocabulary tokens used · 1 source · 1 quote
Residual-quantized trajectory primitives used · 1 source · 1 quote
Reusing image tokens for video frames used · 1 source · 1 quote
RVQ-based speech representation used · 1 source · 1 quote
Separate robotics-specific token weights used · 1 source · 1 quote
Shared frozen encoders used · 1 source · 1 quote
Shared vision backbone used · 1 source · 1 quote
Single-matmul vision projection used · 1 source · 1 quote
Static third-person camera input used · 1 source · 1 quote
Two-layer MLP vision projector used · 1 source · 1 quote
Universal Speech Model-based audio encoder used · 1 source · 1 quote
Variable aspect-ratio image handling used · 1 source · 1 quote
Video-frame pruning used · 1 source · 1 quote
Autoregressive depth-token prefix conditioning optional · 1 source · 1 quote
Depth injection optional · 1 source · 1 quote
Global thumbnail alongside tiled image inputs optional · 1 source · 1 quote
Video preprocessor longest-edge configuration optional · 1 source · 1 quote
Action-modality isolation evaluated · 1 source · 1 quote
Hidden-state conditioning evaluated · 1 source · 1 quote
HPT-style domain-specific input projection evaluated · 1 source · 1 quote
Multimodal Diffusion Transformer decoder evaluated · 1 source · 1 quote
Natural-language hardware-configuration prompts evaluated · 1 source · 1 quote
Per-head per-layer key-value conditioning evaluated · 1 source · 1 quote
Single SigLIP vision encoder evaluated · 1 source · 1 quote
Triplane-based multi-camera tokenization evaluated · 1 source · 1 quote
Vision-and-language navigation evaluated · 1 source · 1 quote

By model

Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

Modelmethods
DeepSeek-V4.1-Flash-1017 coreJoint vision-language pre-training from the start core—
GLM-5.3-Flash usedReusing image tokens for video frames used—
DeepSeek-V4.1-Flash coreJoint vision-language pre-training from the start core3×3 PixelUnshuffle downsampling usedTwo-layer MLP vision projector used—
Step-5-Preview coreMultimodal input understanding core—
DeepSeek-V4-Flash-0731 coreNative multimodal architecture core—
MiMo-V2.6-Flash coreFour-frame audio patches with within-patch bidirectional attention coreJoint multimodal input understanding coreTwo-stage audio encoding with tokenization and patch encoding coreData-parallel-first multimodal encoding usedJoint full-model multimodal training usedTwo-stage text-then-multimodal pre-training used—
Alpamayo 1.5 / 2 Super coreChain-of-thought reasoning vision-language-action model coreSeparate flow-matching action expert coreEnd-to-end autonomous driving usedTriplane-based multi-camera tokenization evaluated—
DeepSeek-V4-Flash-Vision-Exp coreNative multimodal architecture coreVision encoder and multimodal aligner coreVisual modules for multimodal understanding core—
DeepSeek-V4-Pro-0813 coreNative multimodal architecture core—
FLUX 3 Action coreEmbodiment-specific action heads coreRobot-specific input and output layers coreShared action trunk coreShared frozen encoders used—
GR00T N1.7 coreCross-embodiment modeling coreDual-system compositional architecture coreEmbodiment-indexed proprioception MLP coreEmbodiment-specific state and action projectors corePer-embodiment action MLPs coreSigLIP2 pretrained vision transformer coreT5 pretrained text encoder coreVision-language model with diffusion-transformer action head coreMiddle-layer VLM representations for policy conditioning usedPre-trained vision and language encoders used—
Gemma 4 31B coreEncoder-free multimodal architecture coreDedicated multimodal encoders before the LLM usedFrozen encoders during pre-training usedLightweight vision embedding module usedRaw audio projection into LLM embedding space usedSingle-matmul vision projection usedUniversal Speech Model-based audio encoder usedVariable aspect-ratio image handling usedConfigurable visual token budget optional—
Inkling coreDiscrete audio token encoding coreHierarchical patch encoder coreJoint multimodal decoding in a shared hidden space coreNative multimodal architecture coreEncoder-free multimodal architecture used—
Kimi K3 coreNative multimodal architecture coreDecoupled Encoder Process (DEP) usedDual-format coordinate supervision usedFrom-scratch MoonViT-V2 training with next-token prediction usedMoonViT-V2 usedMultimodal input usedPixelShuffle 2×2 visual-token downsampling used—
LFM3-1.7B coreCausal audio encoder with sliding-window attention and short convolutions coreConvolution-attention vision encoder coreDiscrete Residual Vector Quantization audio generation coreDual proprioception conditioning coreEight-codebook Mimi RVQ codes corePairwise sampled-frame embedding coreThree-stage multimodal training protocol coreVision-language extension with feature connector coreASR-based connector alignment usedConnector pre-training with frozen encoders usedDynamic image tiling usedFixed 6:12 text-to-audio token interleaving usedParallel action prediction usedPixelUnshuffle visual-token downsampling usedRepurposed navigation vocabulary tokens usedResidual-quantized trajectory primitives usedSigLIP2 vision encoder with dynamic tiling and token reduction usedSingle-pass continuous action chunk prediction usedVisual route following with target and waypoint prediction usedGlobal thumbnail alongside tiled image inputs optionalVision-and-language navigation evaluated—
LingBot-VLA 2.0 coreConvStack coreHierarchical convolutional decoder coreLearnable image queries corePerceptual-query token appending coreSeparate RGB and depth patch embeddings coreDINOv2 and Point Transformer grasp-policy conditioning usedDepth injection optional—
MiMo-V2.5 coreNative omnimodal architecture coreNative visual and audio understanding coreDedicated visual and audio encoders with lightweight projectors usedMiMo-Audio-initialized audio encoder usedProjector warmup usedVision Transformer (ViT) used—
MiMo-V2.5-Pro coreNative multimodal architecture coreNative omnimodal architecture coreNative visual and audio understanding coreDedicated visual and audio encoders with lightweight projectors usedProjector warmup used—
MiMo-V2.6-Pro coreFour-frame audio patches with within-patch bidirectional attention coreTwo-stage audio encoding with tokenization and patch encoding coreData-parallel-first multimodal encoding usedJoint full-model multimodal training usedTwo-stage text-then-multimodal pre-training used—
MiniMax-M3 coreMixed-modality training from the start core—
Mistral-Large-4 coreMultimodal input coreNative multimodal architecture corePixtral vision encoder corePixtral-style variable-resolution vision encoder coreSpatial patch-merge projector core—
MolmoAct 2 coreContinuous DiT-style action expert coreDepth-aware perception tokens coreMasking action tokens from expert conditioning coreMulti-layer ViT feature concatenation coreMulti-scale image cropping corePer-layer key-value conditioning coreSingle resized crop per image coreTime-conditioned DiT modulation coreGrid quantization of depth maps usedIndex-token marking for multi-image inputs usedLearned image-padding-status embeddings usedMolmo2-ER backbone usedNear-zero action-expert output initialization usedAutoregressive depth-token prefix conditioning optionalHidden-state conditioning evaluatedPer-head per-layer key-value conditioning evaluated—
NVIDIA-Nemotron-3.5-Super-VL-120B-A12B coreC-RADIO vision encoder coreVision-language connector coreVideo-frame pruning used—
OpenVLA / OpenVLA-OFT coreContinuous action representation coreFused SigLIP-DINOv2 vision encoder coreMulti-view image input coreRepresenting robot actions as language-model tokens coreRobot proprioceptive-state projection into language embedding space coreFiLM usedMulti-resolution visual feature fusion usedPatch-as-token vision-language architecture usedShared-sequence multimodal input projection usedStatic third-person camera input usedSingle SigLIP vision encoder evaluated—
Qwen3.5-397B-A17B coreEarly-fusion multimodal training coreThinker–Talker architecture core128-channel mel-spectrogram audio front end usedAudio Transformer (AuT) usedCausal streaming ConvNet codec decoder usedExplicit text-string timestamps usedRVQ-based speech representation usedVideo preprocessor longest-edge configuration optional—
Qwen3.6-35B-A3B coreEarly-fusion multimodal training core—
SmolVLA coreIntermediate VLM-layer features coreReduced action-expert hidden size coreSmall pretrained VLM backbone coreVision-language model backbone coreVisual token reduction coreGlobal-image-only runtime input defaultIntermediate-layer-limited action-expert conditioning defaultSensorimotor-state projection into VLM token space evaluated—
Step-3.7-Flash coreLanguage backbone coreVision encoder coreVision Transformer (ViT) used—
VLA Foundry (Foundry-VLA-1.7B) coreLast-four-layer action-head conditioning corePretrained Qwen3-VL backbone initialization used—
X-VLA corePer-hardware-configuration learnable prompt embeddings coreStreamlined encoding pipeline with soft prompts coreCross-attention-based resampler usedDomain-specific resamplers and action heads usedEmbodiment-aware language prompt concatenation usedPretrained vision-language encoder usedShared vision backbone usedAction-modality isolation evaluatedHPT-style domain-specific input projection evaluatedMultimodal Diffusion Transformer decoder evaluatedNatural-language hardware-configuration prompts evaluated—
Xiaomi-Robotics-1 usedMasking action-related tokens from DiT attention used—
π0.5 coreRouting image and language tokens to the VLM backbone coreRouting robotics-specific inputs to an action expert coreSeparate flow-matching action expert coreMasking missing robot image slots usedSeparate robotics-specific token weights used—
↑