Open Models Almanac
Methodsmodel architecturemultimodal architecture

single mention · model architecture

Vision-language model backbone

A vision-language model backbone encodes multimodal inputs such as images, language instructions, and sensorimotor states into contextual features.1 It combines a vision encoder with a language decoder, and its decoder layers process concatenated image, language, and state tokens before passing the features downstream.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. The VLM encodes these into contextual features, which condition the action expert to generate a continuous sequence of actions — SmolVLA: Efficient Vision-Language-Action Model trained on Lerobot Community Data
  2. The decoder layers process concatenated image, language, and state tokens. The resulting features are then passed to the action expert — SmolVLA: Efficient Vision-Language-Action Model trained on Lerobot Community Data

Also called Vision-Language Model (VLM).

Filed under model architecture › multimodal architecture