Open Models Almanac
Methodsmodel architecturetoken mixersoftmax attention

specific method · filed under model architecture

Causal self-attention

Causal self-attention is a mechanism that allows action tokens to attend to each other within the action expert's architecture.1 It employs a causal attention mask to ensure that each action token can only attend to past tokens within the chunk, preventing future action dependencies.2

Trade-off Interleaving CA and SA layers provides higher success rates and faster inference time.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. SA layers allow the action tokens in to attend to each other — SmolVLA: A vision-language-action model for affordable and efficient robotics
  2. We employ a causal attention mask for the SA layers, ensuring that each action token can only attend to past tokens within the chunk, preventing future action dependencies — SmolVLA: A vision-language-action model for affordable and efficient robotics
  3. Empirically, we find that interleaving CA and SA layers provides higher success rates and faster inference time — SmolVLA: A vision-language-action model for affordable and efficient robotics

Also called Causal self-attention among action tokens, Self-attention (SA), causal self-attention (SA).

sources
2
model
1
lab adopt it
1
strongest
core

Documented in

Hugging Face

Evidence

2 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
evaluated 1core 1

SmolVLA

Hugging Faceadopted2 sources

Self-attention (SA), where action tokens attend to each other (causally—only to the past)

coremodel architecturein SmolVLAHugging Face
1 more from this model

In the SA setting, action tokens attend to each other using a causal mask

evaluatedunclearin SmolVLA
↑