Open Models Almanac
Methodsmodel architecturetoken mixer

specific method · filed under model architecture

Cross-attention

Cross-attention is an attention mechanism in which one set of tokens queries a different set of tokens that serves as the keys and values.1 It lets the action expert's tokens condition on VLM features rather than only attending to each other.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. in the CA setting the VLM features act as keys and values for attention — SmolVLA: A vision-language-action model for affordable and efficient robotics
  2. CA layers cross-attend the VLM’s keys and values, while SA layers allow the action tokens in to attend to each other — SmolVLA: A vision-language-action model for affordable and efficient robotics

Also called simple cross-attention mechanism, cross-attention (CA).

sources
3
models
2
labs adopt it
2
strongest
core

Documented in

Hugging Face

Evidence

3 quotes from 3 sources, one card per model, each opening with its strongest.

How sources treat it
evaluated 1core 2

SmolVLA

Hugging Faceadopted2 sources

Cross-attention (CA), where action tokens attend to the VLM’s features

coremodel architecturein SmolVLAHugging Face
1 more from this model

in the CA setting the VLM features act as keys and values for attention in the action expert.

evaluatedunclearin SmolVLA
↑