SmolVLA
Hugging FaceSelf-attention (SA), where action tokens attend to each other (causally—only to the past)
1 more from this model
In the SA setting, action tokens attend to each other using a causal mask
specific method · filed under model architecture
Causal self-attention is a mechanism that allows action tokens to attend to each other within the action expert's architecture.1 It employs a causal attention mask to ensure that each action token can only attend to past tokens within the chunk, preventing future action dependencies.2
Trade-off Interleaving CA and SA layers provides higher success rates and faster inference time.3
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
SA layers allow the action tokens in to attend to each other— SmolVLA: A vision-language-action model for affordable and efficient robotics
We employ a causal attention mask for the SA layers, ensuring that each action token can only attend to past tokens within the chunk, preventing future action dependencies— SmolVLA: A vision-language-action model for affordable and efficient robotics
Empirically, we find that interleaving CA and SA layers provides higher success rates and faster inference time— SmolVLA: A vision-language-action model for affordable and efficient robotics
Also called Causal self-attention among action tokens, Self-attention (SA), causal self-attention (SA).
2 quotes from 2 sources, one card per model, each opening with its strongest.
Self-attention (SA), where action tokens attend to each other (causally—only to the past)
In the SA setting, action tokens attend to each other using a causal mask